Jul 09 22:30:43.773005 ip-10-0-135-80 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 09 22:30:43.773200 ip-10-0-135-80 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 09 22:30:43.773216 ip-10-0-135-80 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 09 22:30:43.773581 ip-10-0-135-80 systemd[1]: Failed to start Kubernetes Kubelet. Jul 09 22:30:54.011431 ip-10-0-135-80 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 09 22:30:54.011451 ip-10-0-135-80 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot b434a824f203464da8c994925234a2a9 -- Jul 09 22:32:55.330832 ip-10-0-135-80 systemd[1]: Starting Kubernetes Kubelet... Jul 09 22:32:55.797479 ip-10-0-135-80 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 22:32:55.797479 ip-10-0-135-80 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 22:32:55.797479 ip-10-0-135-80 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 22:32:55.797479 ip-10-0-135-80 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 22:32:55.797479 ip-10-0-135-80 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 22:32:55.797479 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.739448 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 22:32:55.797479 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746084 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 22:32:55.797479 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746110 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 22:32:55.797479 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746116 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 22:32:55.797479 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746122 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 22:32:55.797479 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746128 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 22:32:55.797479 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746133 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 22:32:55.797479 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746138 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 22:32:55.797479 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746143 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746147 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746152 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746157 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746161 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746165 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746170 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746174 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746179 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746184 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746188 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746193 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746198 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746201 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746206 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746210 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746214 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746218 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746222 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746226 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 22:32:55.800019 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746230 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746234 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746238 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746242 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746248 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746253 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746257 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746261 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746265 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746271 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746275 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746279 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746283 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746290 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746296 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746300 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746304 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746308 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746312 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746316 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 22:32:55.800948 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746320 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 22:32:55.807485 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746325 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 22:32:55.807485 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746330 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 22:32:55.807485 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746334 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 22:32:55.807485 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746339 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 22:32:55.807485 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746343 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 22:32:55.807485 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746347 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 22:32:55.807485 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746351 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 22:32:55.807485 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746355 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 22:32:55.807485 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746359 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 22:32:55.807485 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746364 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 22:32:55.807485 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746368 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 22:32:55.807485 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746372 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 22:32:55.807485 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746379 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 22:32:55.807485 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746384 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 22:32:55.807485 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746389 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 22:32:55.807485 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746393 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 22:32:55.807485 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746397 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 22:32:55.807485 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746402 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 22:32:55.807485 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746406 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746410 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746414 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746418 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746422 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746426 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746431 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746434 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746439 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746443 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746447 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746451 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746455 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746459 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746463 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746468 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746471 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746475 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746479 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746483 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 22:32:55.808396 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746487 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 22:32:55.809482 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746491 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 22:32:55.809482 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746496 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 09 22:32:55.809482 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746501 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 22:32:55.809482 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746505 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 22:32:55.809482 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746509 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 22:32:55.809482 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746513 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 22:32:55.809482 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746518 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 22:32:55.809482 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746524 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 22:32:55.809482 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746531 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 22:32:55.809482 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746537 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 22:32:55.809482 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746542 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 22:32:55.809482 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.746547 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 22:32:55.809482 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747713 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 22:32:55.809482 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747723 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 22:32:55.809482 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747728 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 22:32:55.809482 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747733 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 22:32:55.809482 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747737 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 22:32:55.809482 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747742 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 22:32:55.809482 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747746 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747751 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747756 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747760 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747765 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747769 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747773 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747779 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747783 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747787 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747791 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747795 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747799 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747803 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747807 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747812 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747816 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747820 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747825 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747829 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 22:32:55.812994 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747833 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747840 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747846 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747851 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747855 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747860 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747864 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747869 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747873 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747876 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747881 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747885 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747889 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747893 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747897 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747901 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747905 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747910 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747915 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747922 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 22:32:55.813711 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747926 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747930 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747934 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747938 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747941 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747945 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747949 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747988 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747994 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.747999 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748004 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748008 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748012 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748017 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748022 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748026 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748030 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748034 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748039 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748043 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 22:32:55.814365 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748046 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 22:32:55.815221 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748050 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 22:32:55.815221 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748054 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 22:32:55.815221 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748060 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 22:32:55.815221 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748065 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 22:32:55.815221 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748069 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 22:32:55.815221 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748073 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 22:32:55.815221 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748077 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 22:32:55.815221 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748081 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 22:32:55.815221 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748086 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 22:32:55.815221 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748090 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 22:32:55.815221 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748095 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 22:32:55.815221 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748099 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 22:32:55.815221 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748103 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 22:32:55.815221 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748107 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 22:32:55.815221 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748111 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 22:32:55.815221 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748116 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 22:32:55.815221 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748120 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 22:32:55.815221 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748124 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 22:32:55.815221 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748130 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 22:32:55.816031 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748135 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 22:32:55.816031 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748139 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 22:32:55.816031 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748145 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 22:32:55.816031 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748151 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 22:32:55.816031 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748156 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 22:32:55.816031 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748160 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 22:32:55.816031 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748164 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 22:32:55.816031 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748170 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 22:32:55.816031 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748174 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 22:32:55.816031 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748178 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 22:32:55.816031 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748182 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 22:32:55.816031 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748186 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 22:32:55.816031 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748190 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 22:32:55.816031 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748810 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 22:32:55.816031 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748818 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 22:32:55.816031 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748823 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 22:32:55.816031 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748827 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 22:32:55.816031 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748831 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 22:32:55.816031 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748835 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 22:32:55.816664 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748841 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 22:32:55.816664 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748847 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 22:32:55.816664 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748852 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 22:32:55.816664 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748856 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 22:32:55.816664 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748861 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 22:32:55.816664 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748866 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 22:32:55.816664 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748870 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 22:32:55.816664 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748874 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 22:32:55.816664 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748878 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 22:32:55.816664 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748883 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 22:32:55.816664 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748887 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 22:32:55.816664 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748891 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 22:32:55.816664 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748895 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 22:32:55.816664 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748899 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 22:32:55.816664 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748903 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 22:32:55.816664 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748907 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 22:32:55.816664 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748912 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 22:32:55.816664 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748916 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 22:32:55.816664 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748920 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 22:32:55.987211 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748925 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 22:32:55.987211 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748930 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 22:32:55.987211 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748934 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 22:32:55.987211 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748938 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 22:32:55.987211 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748942 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 22:32:55.987211 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748947 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 22:32:55.987211 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748951 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 22:32:55.987211 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748956 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 22:32:55.987211 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748960 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 22:32:55.987211 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748964 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 22:32:55.987211 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748968 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 22:32:55.987211 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748972 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 22:32:55.987211 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748976 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 22:32:55.987211 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748980 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 22:32:55.987211 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748984 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 22:32:55.987211 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748989 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 22:32:55.987211 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748993 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 22:32:55.987211 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.748997 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 22:32:55.987211 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749001 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 22:32:55.840311 ip-10-0-135-80 systemd[1]: Started Kubernetes Kubelet. Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749005 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749009 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749013 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749017 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749021 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749025 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749031 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749036 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749040 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749046 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749052 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749057 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749061 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749066 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749070 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749075 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749079 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749083 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749087 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 22:32:55.999144 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749095 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749099 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749103 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749108 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749112 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749116 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749120 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749124 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749128 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749132 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749136 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749141 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749145 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749149 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749154 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749158 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749161 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749166 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749170 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749173 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 22:32:56.001183 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749177 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749181 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749186 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749191 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749195 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749199 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749203 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749207 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749211 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749216 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749220 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749225 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749233 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749237 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749241 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.749246 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749333 2568 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749345 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749354 2568 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749361 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749369 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 22:32:56.002101 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749374 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749381 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749388 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749393 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749397 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749403 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749408 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749414 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749419 2568 flags.go:64] FLAG: --cgroup-root="" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749423 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749428 2568 flags.go:64] FLAG: --client-ca-file="" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749436 2568 flags.go:64] FLAG: --cloud-provider="external" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749441 2568 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749449 2568 flags.go:64] FLAG: --cluster-domain="" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749454 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749459 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749464 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749470 2568 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749475 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749480 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749485 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749491 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749496 2568 flags.go:64] FLAG: --contention-profiling="false" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749503 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 22:32:56.003964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749507 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749512 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749517 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749524 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749529 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749533 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749538 2568 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749543 2568 flags.go:64] FLAG: --enable-server="true" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749548 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749554 2568 flags.go:64] FLAG: --event-burst="100" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749560 2568 flags.go:64] FLAG: --event-qps="50" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749585 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749590 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749595 2568 flags.go:64] FLAG: --eviction-hard="" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749601 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749607 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749611 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749616 2568 flags.go:64] FLAG: --eviction-soft="" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749621 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749628 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749632 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749637 2568 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749641 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749646 2568 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749651 2568 flags.go:64] FLAG: --feature-gates="" Jul 09 22:32:56.004850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749657 2568 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749661 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749666 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749671 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749676 2568 flags.go:64] FLAG: --healthz-port="10248" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749681 2568 flags.go:64] FLAG: --help="false" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749698 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-135-80.ec2.internal" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749704 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749708 2568 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749713 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749719 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749724 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749728 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749735 2568 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749739 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749744 2568 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749749 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749754 2568 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749759 2568 flags.go:64] FLAG: --kube-reserved="" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749763 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749768 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749773 2568 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749777 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749782 2568 flags.go:64] FLAG: --lock-file="" Jul 09 22:32:56.005812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749786 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749791 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749797 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749807 2568 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749811 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749816 2568 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749820 2568 flags.go:64] FLAG: --logging-format="text" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749825 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749831 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749835 2568 flags.go:64] FLAG: --manifest-url="" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749840 2568 flags.go:64] FLAG: --manifest-url-header="" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749847 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749852 2568 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749859 2568 flags.go:64] FLAG: --max-pods="110" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749866 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749871 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749875 2568 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749881 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749885 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749890 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749895 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749906 2568 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749910 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749915 2568 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 22:32:56.006927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749920 2568 flags.go:64] FLAG: --pod-cidr="" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749925 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749934 2568 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749939 2568 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749944 2568 flags.go:64] FLAG: --pods-per-core="0" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749949 2568 flags.go:64] FLAG: --port="10250" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749953 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749958 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f713cc0ee360e498" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749964 2568 flags.go:64] FLAG: --qos-reserved="" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749968 2568 flags.go:64] FLAG: --read-only-port="10255" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749974 2568 flags.go:64] FLAG: --register-node="true" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749979 2568 flags.go:64] FLAG: --register-with-taints="" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749985 2568 flags.go:64] FLAG: --registry-burst="10" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749989 2568 flags.go:64] FLAG: --registry-qps="5" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749994 2568 flags.go:64] FLAG: --reserved-cpus="" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.749998 2568 flags.go:64] FLAG: --reserved-memory="" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750004 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750009 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750014 2568 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750019 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750024 2568 flags.go:64] FLAG: --runonce="false" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750029 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750034 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750042 2568 flags.go:64] FLAG: --seccomp-default="false" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750047 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 22:32:56.007814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750052 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750056 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750062 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750067 2568 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750071 2568 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750076 2568 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750081 2568 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750085 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750090 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750095 2568 flags.go:64] FLAG: --system-cgroups="" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750099 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750108 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750113 2568 flags.go:64] FLAG: --tls-cert-file="" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750117 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750124 2568 flags.go:64] FLAG: --tls-min-version="" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750129 2568 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750133 2568 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750140 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750145 2568 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750149 2568 flags.go:64] FLAG: --v="2" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750156 2568 flags.go:64] FLAG: --version="false" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750167 2568 flags.go:64] FLAG: --vmodule="" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750188 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750193 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750466 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 22:32:56.008752 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750475 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 22:32:56.009558 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750480 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 22:32:56.009558 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750484 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 22:32:56.009558 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750488 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 22:32:56.009558 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750492 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 22:32:56.009558 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750499 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 22:32:56.009558 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750510 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 22:32:56.009558 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750515 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 22:32:56.009558 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750519 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 22:32:56.009558 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750523 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 22:32:56.009558 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750527 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 22:32:56.009558 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750531 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 22:32:56.009558 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750536 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 22:32:56.009558 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750540 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 22:32:56.009558 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750544 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 22:32:56.009558 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750548 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 22:32:56.009558 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750552 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 22:32:56.009558 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750556 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 22:32:56.009558 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750577 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 22:32:56.009558 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750583 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750590 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750594 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750598 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750602 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750607 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750613 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750617 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750622 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750626 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750630 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750634 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750638 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750642 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750647 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750651 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750655 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750659 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750665 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750670 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 22:32:56.010264 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750681 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 22:32:56.010975 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750686 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 22:32:56.010975 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750690 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 22:32:56.010975 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750694 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 22:32:56.010975 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750699 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 22:32:56.010975 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750703 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 22:32:56.010975 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750707 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 22:32:56.010975 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750711 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 22:32:56.010975 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750715 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 22:32:56.010975 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750719 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 22:32:56.010975 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750723 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 22:32:56.010975 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750728 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 22:32:56.010975 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750731 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 22:32:56.010975 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750735 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 22:32:56.010975 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750740 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 22:32:56.010975 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750744 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 22:32:56.010975 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750748 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 22:32:56.010975 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750752 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 22:32:56.010975 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750758 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 22:32:56.010975 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750764 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750769 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750774 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750779 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750783 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750788 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750791 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750795 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750800 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750803 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750807 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750814 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750818 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750822 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750837 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750841 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750845 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750849 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750854 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750858 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 22:32:56.011679 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750862 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 22:32:56.012275 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750866 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 22:32:56.012275 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750870 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 22:32:56.012275 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750874 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 22:32:56.012275 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750878 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 22:32:56.012275 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750882 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 22:32:56.012275 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750886 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 22:32:56.012275 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750890 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 22:32:56.012275 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750894 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 22:32:56.012275 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750900 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 22:32:56.012275 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750905 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 22:32:56.012275 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750912 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 22:32:56.012275 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750917 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 22:32:56.012275 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750921 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 22:32:56.012275 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750926 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 22:32:56.012275 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750935 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 22:32:56.012275 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750939 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 22:32:56.012275 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750944 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 22:32:56.012275 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750948 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 22:32:56.012275 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.750951 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 22:32:56.012907 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.750960 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 22:32:56.012907 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.766058 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 09 22:32:56.012907 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.766085 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 22:32:56.012907 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.766114 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 22:32:56.012907 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.766121 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 22:32:56.012907 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766193 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 22:32:56.012907 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766199 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 22:32:56.012907 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766202 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 22:32:56.012907 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766206 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 22:32:56.012907 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766209 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 22:32:56.012907 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766212 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 22:32:56.012907 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766215 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 22:32:56.012907 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766218 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 22:32:56.012907 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766221 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 22:32:56.012907 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766225 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766228 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766232 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766235 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766238 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766241 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766245 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766249 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766252 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766256 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766259 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766262 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766265 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766267 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766270 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766273 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766275 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766278 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766281 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766284 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 22:32:56.013476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766287 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 22:32:56.014145 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766289 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 22:32:56.014145 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766292 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 22:32:56.014145 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766295 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 22:32:56.014145 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766298 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 22:32:56.014145 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766301 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 22:32:56.014145 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766304 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 22:32:56.014145 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766307 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 22:32:56.014145 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766309 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 09 22:32:56.014145 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766312 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 22:32:56.014145 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766320 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 22:32:56.014145 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766325 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 22:32:56.014145 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766329 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 22:32:56.014145 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766333 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 22:32:56.014145 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766336 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 22:32:56.014145 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766339 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 22:32:56.014145 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766343 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 22:32:56.014145 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766346 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 22:32:56.014145 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766349 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 22:32:56.014145 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766355 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 22:32:56.014962 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766360 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 22:32:56.014962 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766363 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 22:32:56.014962 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766366 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 22:32:56.014962 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766370 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 22:32:56.014962 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766373 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 22:32:56.014962 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766376 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 22:32:56.014962 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766378 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 22:32:56.014962 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766381 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 22:32:56.014962 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766384 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 22:32:56.014962 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766387 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 22:32:56.014962 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766391 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 22:32:56.014962 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766395 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 22:32:56.014962 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766398 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 22:32:56.014962 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766401 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 22:32:56.014962 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766404 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 22:32:56.014962 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766407 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 22:32:56.014962 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766410 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 22:32:56.014962 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766413 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 22:32:56.014962 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766415 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766418 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766421 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766423 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766428 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766430 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766433 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766436 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766439 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766442 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766444 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766448 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766457 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766462 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766465 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766468 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766471 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766473 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766476 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766479 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 22:32:56.015551 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766482 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 22:32:56.016247 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766485 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 22:32:56.016247 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766488 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 22:32:56.016247 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766491 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 22:32:56.016247 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766494 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 22:32:56.016247 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766497 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 22:32:56.016247 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766499 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 22:32:56.016247 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766502 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 22:32:56.016247 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766505 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 22:32:56.016247 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766508 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 22:32:56.016247 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766511 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 22:32:56.016247 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766513 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 22:32:56.016247 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.766519 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 22:32:56.016247 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766684 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 22:32:56.016247 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766692 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 22:32:56.016247 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766695 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 22:32:56.016801 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766699 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 22:32:56.016801 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766704 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 22:32:56.016801 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766707 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 22:32:56.016801 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766710 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 22:32:56.016801 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766713 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 22:32:56.016801 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766716 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 22:32:56.016801 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766718 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 22:32:56.016801 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766721 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 22:32:56.016801 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766724 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 22:32:56.016801 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766735 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 22:32:56.016801 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766738 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 22:32:56.016801 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766742 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 22:32:56.016801 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766747 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 22:32:56.016801 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766752 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 22:32:56.016801 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766756 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 22:32:56.016801 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766761 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 22:32:56.016801 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766766 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 22:32:56.016801 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766770 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 22:32:56.016801 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766773 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766778 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766782 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766787 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766791 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766795 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766797 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766800 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766803 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766806 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766809 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766812 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766815 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766818 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766821 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766825 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766828 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766831 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766833 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766836 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 22:32:56.017400 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766839 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 22:32:56.017997 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766841 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 22:32:56.017997 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766844 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 22:32:56.017997 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766849 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 22:32:56.017997 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766853 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 22:32:56.017997 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766856 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 22:32:56.017997 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766859 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 22:32:56.017997 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766862 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 22:32:56.017997 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766864 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 22:32:56.017997 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766867 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 22:32:56.017997 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766870 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 22:32:56.017997 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766872 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 22:32:56.017997 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766875 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 22:32:56.017997 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766878 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 22:32:56.017997 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766882 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 22:32:56.017997 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766886 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 22:32:56.017997 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766888 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 22:32:56.017997 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766891 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 22:32:56.017997 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766893 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 22:32:56.017997 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766896 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 22:32:56.018946 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766899 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 22:32:56.018946 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766902 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 22:32:56.018946 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766904 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 22:32:56.018946 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766907 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 22:32:56.018946 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766909 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 22:32:56.018946 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766912 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 22:32:56.018946 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766915 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 22:32:56.018946 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766919 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 22:32:56.018946 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766922 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 22:32:56.018946 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766925 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 22:32:56.018946 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766927 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 22:32:56.018946 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766932 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 22:32:56.018946 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766935 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 22:32:56.018946 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766939 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 22:32:56.018946 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766942 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 22:32:56.018946 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766946 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 22:32:56.018946 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766949 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 22:32:56.018946 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766953 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 22:32:56.018946 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766956 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 22:32:56.108647 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766958 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 22:32:56.108647 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766961 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 22:32:56.108647 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766964 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 22:32:56.108647 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766966 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 22:32:56.108647 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766969 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 22:32:56.108647 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766972 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 22:32:56.108647 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766974 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 22:32:56.108647 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766977 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 22:32:56.108647 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766979 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 22:32:56.108647 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766982 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 09 22:32:56.108647 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766985 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 22:32:56.108647 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766988 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 22:32:56.108647 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766990 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 22:32:56.108647 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766993 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 22:32:56.108647 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766995 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 22:32:56.108647 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.766998 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 22:32:56.108647 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.767002 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 22:32:56.108647 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.767004 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 22:32:56.108647 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:55.767007 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 22:32:56.109360 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.767012 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 22:32:56.109360 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.767184 2568 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 22:32:56.109360 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.769631 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 22:32:56.109360 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.771164 2568 server.go:1019] "Starting client certificate rotation" Jul 09 22:32:56.109360 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.771288 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 22:32:56.109360 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.771327 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 22:32:56.109360 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.774960 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 22:32:56.109360 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.777857 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 22:32:56.109360 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.789336 2568 log.go:25] "Validated CRI v1 runtime API" Jul 09 22:32:56.109360 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.791860 2568 log.go:25] "Validated CRI v1 image API" Jul 09 22:32:56.109360 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.794336 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 22:32:56.109360 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.813218 2568 fs.go:135] Filesystem UUIDs: map[4086e43d-4d83-4b0d-8614-1e1d9674f998:/dev/nvme0n1p3 6bfc6345-8c3c-40e2-aed0-bb974e2e4b58:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 09 22:32:56.109790 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.813245 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 22:32:56.109790 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.815851 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 22:32:56.110052 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.819996 2568 manager.go:216] Machine: {Timestamp:2026-07-09 22:32:55.81852232 +0000 UTC m=+0.359601109 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3048796 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2632f7d10142b19dda335c0686ac92 SystemUUID:ec2632f7-d101-42b1-9dda-335c0686ac92 BootID:b434a824-f203-464d-a8c9-94925234a2a9 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:06:91:1c:85:3d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:06:91:1c:85:3d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:41:47:71:f6:43 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 22:32:56.110052 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.820172 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 22:32:56.110052 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.820308 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 22:32:56.110052 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.823023 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 22:32:56.110052 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.823065 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-80.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 22:32:56.110052 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.823850 2568 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 22:32:56.110052 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.823866 2568 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 22:32:56.110052 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.823886 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 22:32:56.110052 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.824523 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 22:32:56.110052 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.825834 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 22:32:56.110052 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.828620 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 09 22:32:56.110052 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.828780 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 22:32:56.110052 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.828813 2568 kubelet.go:486] "Attempting to sync node with API server" Jul 09 22:32:56.110052 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.828824 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 22:32:56.110052 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.828846 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 22:32:56.110052 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.828857 2568 kubelet.go:398] "Adding apiserver pod source" Jul 09 22:32:56.110052 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.828869 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 22:32:56.110052 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.829240 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.829256 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.830971 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j2vjv" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.831379 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.833585 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.833627 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.834794 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.834817 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.834826 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.834834 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.834850 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.834858 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.834871 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.834879 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.834891 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.834902 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.834923 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.834930 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.835427 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.835435 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.839271 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-80.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:55.839295 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:55.839293 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.839543 2568 server.go:1268] "Started kubelet" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.839901 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j2vjv" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.841453 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.841914 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.842001 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.843682 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.844399 2568 server.go:310] "Adding debug handlers to kubelet server" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:55.847952 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.850810 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.850850 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:55.850948 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.851556 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.851559 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.851600 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.851704 2568 reconstruct.go:97] "Volume reconstruction finished" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.851714 2568 reconciler.go:29] "Reconciler: start to sync state" Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.851843 2568 factory.go:55] Registering systemd factory Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.851857 2568 factory.go:223] Registration of the systemd container factory successfully Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.852116 2568 factory.go:153] Registering CRI-O factory Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.852132 2568 factory.go:223] Registration of the crio container factory successfully Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.852196 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.852226 2568 factory.go:103] Registering Raw factory Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.852243 2568 manager.go:1196] Started watching for new ooms in manager Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.852820 2568 manager.go:349] Starting recovery of all containers Jul 09 22:32:56.110682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.854643 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:55.863967 2568 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-80.ec2.internal\" not found" node="ip-10-0-135-80.ec2.internal" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.869915 2568 manager.go:354] Recovery completed Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.875788 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.878040 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientMemory" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.878080 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.878095 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientPID" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.878713 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.878724 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.878732 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.878746 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.879901 2568 policy_none.go:49] "None policy: Start" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.879934 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.879945 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.880046 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientMemory" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.880075 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.880089 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientPID" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.880820 2568 policy_none.go:47] "Start" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.936456 2568 manager.go:337] "Starting Device Plugin manager" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:55.936531 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.936544 2568 server.go:87] "Starting device plugin registration server" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.936872 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.936887 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.936966 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.937068 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:55.937079 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:55.937737 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:55.937789 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-80.ec2.internal\" not found" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.037548 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.038955 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientMemory" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.038989 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.039000 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientPID" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.039031 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-80.ec2.internal" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.039879 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.041268 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.041300 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.041322 2568 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:56.041355 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.046596 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.048275 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-80.ec2.internal" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:56.048295 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-80.ec2.internal\": node \"ip-10-0-135-80.ec2.internal\" not found" Jul 09 22:32:56.112136 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:56.108898 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.143560 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal"] Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.143667 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.145341 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientMemory" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.145374 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.145385 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientPID" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.146652 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.146780 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.146815 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.147547 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientMemory" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.147554 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientMemory" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.147573 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.147582 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.147586 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientPID" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.147594 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientPID" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.148673 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.148701 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.149504 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientMemory" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.149530 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.149542 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeHasSufficientPID" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.153492 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acfe4b356a98be4a5be8184b12458310-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" (UID: \"acfe4b356a98be4a5be8184b12458310\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.153527 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4bd33866a0efa68864d83a51f0f716fd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-80.ec2.internal\" (UID: \"4bd33866a0efa68864d83a51f0f716fd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.153557 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/acfe4b356a98be4a5be8184b12458310-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" (UID: \"acfe4b356a98be4a5be8184b12458310\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:56.171452 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-80.ec2.internal\" not found" node="ip-10-0-135-80.ec2.internal" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:56.176483 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-80.ec2.internal\" not found" node="ip-10-0-135-80.ec2.internal" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:56.210141 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.254083 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/acfe4b356a98be4a5be8184b12458310-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" (UID: \"acfe4b356a98be4a5be8184b12458310\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.254119 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acfe4b356a98be4a5be8184b12458310-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" (UID: \"acfe4b356a98be4a5be8184b12458310\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Jul 09 22:32:56.297024 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.254139 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4bd33866a0efa68864d83a51f0f716fd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-80.ec2.internal\" (UID: \"4bd33866a0efa68864d83a51f0f716fd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" Jul 09 22:32:56.299100 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.254191 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4bd33866a0efa68864d83a51f0f716fd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-80.ec2.internal\" (UID: \"4bd33866a0efa68864d83a51f0f716fd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" Jul 09 22:32:56.299100 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.254206 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acfe4b356a98be4a5be8184b12458310-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" (UID: \"acfe4b356a98be4a5be8184b12458310\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Jul 09 22:32:56.299100 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.254206 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/acfe4b356a98be4a5be8184b12458310-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal\" (UID: \"acfe4b356a98be4a5be8184b12458310\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Jul 09 22:32:56.487982 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:56.310885 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Jul 09 22:32:56.487982 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:56.415374 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Jul 09 22:32:56.487982 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.475155 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Jul 09 22:32:56.487982 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.479312 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" Jul 09 22:32:56.677961 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:56.516617 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Jul 09 22:32:56.677961 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:56.616952 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Jul 09 22:32:56.900427 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:56.718563 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Jul 09 22:32:56.900427 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.770393 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 22:32:56.900427 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.770570 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 22:32:56.900427 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.770571 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 22:32:56.900427 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:56.818861 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-80.ec2.internal\" not found" Jul 09 22:32:56.900427 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.844547 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 22:27:56 +0000 UTC" deadline="2028-02-02 17:40:37.165047508 +0000 UTC" Jul 09 22:32:56.900427 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.844595 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13747h7m40.320455177s" Jul 09 22:32:56.900427 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.858690 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 22:32:57.093610 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.949001 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 22:32:57.093610 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.951690 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" Jul 09 22:32:57.093610 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.963793 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 22:32:57.093610 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.964960 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" Jul 09 22:32:57.093610 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:56.976321 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 22:32:57.429052 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:57.367253 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacfe4b356a98be4a5be8184b12458310.slice/crio-386d9f4fb0d2b3b1ae9da5db74be940a7b286c862cf39eefe5e7fb06dc81a494 WatchSource:0}: Error finding container 386d9f4fb0d2b3b1ae9da5db74be940a7b286c862cf39eefe5e7fb06dc81a494: Status 404 returned error can't find the container with id 386d9f4fb0d2b3b1ae9da5db74be940a7b286c862cf39eefe5e7fb06dc81a494 Jul 09 22:32:57.429052 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:57.370720 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd33866a0efa68864d83a51f0f716fd.slice/crio-6924dae4f00c111597071421f3e75f40fdd28a086a0a289d610b2abfb052306d WatchSource:0}: Error finding container 6924dae4f00c111597071421f3e75f40fdd28a086a0a289d610b2abfb052306d: Status 404 returned error can't find the container with id 6924dae4f00c111597071421f3e75f40fdd28a086a0a289d610b2abfb052306d Jul 09 22:32:57.429052 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.378839 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 22:32:57.668344 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.668309 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 22:32:57.832889 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.832779 2568 apiserver.go:52] "Watching apiserver" Jul 09 22:32:57.846476 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.844765 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 22:32:57.846476 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.845741 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-8ddvw","openshift-multus/multus-c8lxb","kube-system/konnectivity-agent-6vrvs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2","openshift-multus/network-metrics-daemon-5wch9","openshift-network-diagnostics/network-check-target-fvsts","openshift-network-operator/iptables-alerter-pwnxw","openshift-ovn-kubernetes/ovnkube-node-pm8p2","kube-system/global-pull-secret-syncer-9nmjj","kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal","openshift-cluster-node-tuning-operator/tuned-8nxpc","openshift-dns/node-resolver-sgpbm","openshift-image-registry/node-ca-vsrl6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal"] Jul 09 22:32:57.848650 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.848623 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pwnxw" Jul 09 22:32:57.850929 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.850906 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vsrl6" Jul 09 22:32:57.851549 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.851529 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 22:32:57.852008 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.851907 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 22:32:57.852375 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.852352 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 22:32:57.852542 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.852457 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 22:32:57.852635 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.852585 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-q6vwn\"" Jul 09 22:32:57.853720 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.853386 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6vrvs" Jul 09 22:32:57.853850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.853819 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 22:32:57.854164 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.854065 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 22:32:57.854164 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.854087 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7fglt\"" Jul 09 22:32:57.854164 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.854106 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 22:32:57.856990 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.856915 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-srd44\"" Jul 09 22:32:57.856990 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.856967 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 22:32:57.857174 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.857036 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 22:32:57.857251 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.857233 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:57.860047 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.860025 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 22:32:57.860879 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.860539 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qzr4z\"" Jul 09 22:32:57.860879 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.860543 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 22:32:57.860879 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.860544 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 22:32:57.860879 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.860709 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:32:57.860879 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:57.860811 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:32:57.864008 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.863987 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 22:32:57.864947 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.864921 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsx7h\" (UniqueName: \"kubernetes.io/projected/b8d3fdbc-afb4-486f-a376-b4e70e32bdd8-kube-api-access-hsx7h\") pod \"node-ca-vsrl6\" (UID: \"b8d3fdbc-afb4-486f-a376-b4e70e32bdd8\") " pod="openshift-image-registry/node-ca-vsrl6" Jul 09 22:32:57.865112 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.864958 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b8d3fdbc-afb4-486f-a376-b4e70e32bdd8-serviceca\") pod \"node-ca-vsrl6\" (UID: \"b8d3fdbc-afb4-486f-a376-b4e70e32bdd8\") " pod="openshift-image-registry/node-ca-vsrl6" Jul 09 22:32:57.865112 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.864986 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5eb6836b-6d35-49c3-899b-d9e81847d4fb-agent-certs\") pod \"konnectivity-agent-6vrvs\" (UID: \"5eb6836b-6d35-49c3-899b-d9e81847d4fb\") " pod="kube-system/konnectivity-agent-6vrvs" Jul 09 22:32:57.865112 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.865000 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:32:57.865112 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.865010 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5eb6836b-6d35-49c3-899b-d9e81847d4fb-konnectivity-ca\") pod \"konnectivity-agent-6vrvs\" (UID: \"5eb6836b-6d35-49c3-899b-d9e81847d4fb\") " pod="kube-system/konnectivity-agent-6vrvs" Jul 09 22:32:57.865112 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:57.865076 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvsts" podUID="9602778d-6209-402b-8ccf-4267f9a37ad5" Jul 09 22:32:57.865432 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.865107 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c341c17b-674a-4de7-a6a3-a428f3a655e3-iptables-alerter-script\") pod \"iptables-alerter-pwnxw\" (UID: \"c341c17b-674a-4de7-a6a3-a428f3a655e3\") " pod="openshift-network-operator/iptables-alerter-pwnxw" Jul 09 22:32:57.865432 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.865146 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c341c17b-674a-4de7-a6a3-a428f3a655e3-host-slash\") pod \"iptables-alerter-pwnxw\" (UID: \"c341c17b-674a-4de7-a6a3-a428f3a655e3\") " pod="openshift-network-operator/iptables-alerter-pwnxw" Jul 09 22:32:57.865432 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.865191 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frxwl\" (UniqueName: \"kubernetes.io/projected/c341c17b-674a-4de7-a6a3-a428f3a655e3-kube-api-access-frxwl\") pod \"iptables-alerter-pwnxw\" (UID: \"c341c17b-674a-4de7-a6a3-a428f3a655e3\") " pod="openshift-network-operator/iptables-alerter-pwnxw" Jul 09 22:32:57.865432 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.865221 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8d3fdbc-afb4-486f-a376-b4e70e32bdd8-host\") pod \"node-ca-vsrl6\" (UID: \"b8d3fdbc-afb4-486f-a376-b4e70e32bdd8\") " pod="openshift-image-registry/node-ca-vsrl6" Jul 09 22:32:57.867610 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.867587 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:32:57.867721 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:57.867684 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nmjj" podUID="44657019-d54e-4ff7-8677-c4ec3d34e1f8" Jul 09 22:32:57.870263 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.870239 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:57.872920 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.872889 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c8lxb" Jul 09 22:32:57.873197 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.873172 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 22:32:57.873839 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.873632 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 22:32:57.873839 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.873644 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h7zb4\"" Jul 09 22:32:57.873839 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.873638 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 22:32:57.873839 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.873637 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 22:32:57.873839 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.873727 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 22:32:57.875939 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.875694 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.875939 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.875815 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 22:32:57.876329 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.876312 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2tnxd\"" Jul 09 22:32:57.878562 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.878525 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 22:32:57.879071 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.878903 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 22:32:57.879071 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.878960 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 22:32:57.879071 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.879070 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:57.879608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.879588 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5h5bm\"" Jul 09 22:32:57.879716 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.879627 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 22:32:57.880253 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.880231 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 22:32:57.880340 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.880233 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 22:32:57.881767 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.881748 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sgpbm" Jul 09 22:32:57.882037 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.882009 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 22:32:57.882211 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.882188 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 22:32:57.882211 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.882203 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xlptm\"" Jul 09 22:32:57.883334 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.883311 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zfvb5" Jul 09 22:32:57.885013 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.884860 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 22:32:57.885013 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.884860 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 22:32:57.885239 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.885219 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jjpkp\"" Jul 09 22:32:57.891554 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.891521 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zfvb5" Jul 09 22:32:57.953270 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.953236 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 22:32:57.965940 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.965907 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-run-ovn-kubernetes\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.965940 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.965948 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca43f479-c01f-49e4-b1fe-a36149b29f51-cnibin\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:57.966196 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.965977 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f61fa573-7943-469d-8262-523ad3aabb94-device-dir\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:57.966196 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966026 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b8d3fdbc-afb4-486f-a376-b4e70e32bdd8-serviceca\") pod \"node-ca-vsrl6\" (UID: \"b8d3fdbc-afb4-486f-a376-b4e70e32bdd8\") " pod="openshift-image-registry/node-ca-vsrl6" Jul 09 22:32:57.966196 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966088 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca43f479-c01f-49e4-b1fe-a36149b29f51-cni-binary-copy\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:57.966346 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966191 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca43f479-c01f-49e4-b1fe-a36149b29f51-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:57.966346 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966227 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-lib-modules\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:57.966346 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966253 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85d0d04e-7886-4803-82a9-980ae8b0d714-tmp-dir\") pod \"node-resolver-sgpbm\" (UID: \"85d0d04e-7886-4803-82a9-980ae8b0d714\") " pod="openshift-dns/node-resolver-sgpbm" Jul 09 22:32:57.966346 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966308 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f61fa573-7943-469d-8262-523ad3aabb94-etc-selinux\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:57.966346 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966343 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c341c17b-674a-4de7-a6a3-a428f3a655e3-host-slash\") pod \"iptables-alerter-pwnxw\" (UID: \"c341c17b-674a-4de7-a6a3-a428f3a655e3\") " pod="openshift-network-operator/iptables-alerter-pwnxw" Jul 09 22:32:57.966561 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966384 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsx7h\" (UniqueName: \"kubernetes.io/projected/b8d3fdbc-afb4-486f-a376-b4e70e32bdd8-kube-api-access-hsx7h\") pod \"node-ca-vsrl6\" (UID: \"b8d3fdbc-afb4-486f-a376-b4e70e32bdd8\") " pod="openshift-image-registry/node-ca-vsrl6" Jul 09 22:32:57.966561 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966441 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-host-run-multus-certs\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:57.966561 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966458 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c341c17b-674a-4de7-a6a3-a428f3a655e3-host-slash\") pod \"iptables-alerter-pwnxw\" (UID: \"c341c17b-674a-4de7-a6a3-a428f3a655e3\") " pod="openshift-network-operator/iptables-alerter-pwnxw" Jul 09 22:32:57.966561 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966469 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-run-ovn\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.966561 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966521 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44c0e227-9c8f-4de3-8923-de936db6502c-tmp\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:57.966561 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966551 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws69l\" (UniqueName: \"kubernetes.io/projected/44c0e227-9c8f-4de3-8923-de936db6502c-kube-api-access-ws69l\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:57.966856 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966576 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs\") pod \"network-metrics-daemon-5wch9\" (UID: \"941cb948-d902-4c71-abf2-52ec25863dd8\") " pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:32:57.966856 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966605 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vgdx\" (UniqueName: \"kubernetes.io/projected/c7df5c30-4311-4d4e-ae82-e8822df20edb-kube-api-access-6vgdx\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:57.966856 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966606 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b8d3fdbc-afb4-486f-a376-b4e70e32bdd8-serviceca\") pod \"node-ca-vsrl6\" (UID: \"b8d3fdbc-afb4-486f-a376-b4e70e32bdd8\") " pod="openshift-image-registry/node-ca-vsrl6" Jul 09 22:32:57.966856 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966659 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-host\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:57.966856 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966681 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89zg5\" (UniqueName: \"kubernetes.io/projected/941cb948-d902-4c71-abf2-52ec25863dd8-kube-api-access-89zg5\") pod \"network-metrics-daemon-5wch9\" (UID: \"941cb948-d902-4c71-abf2-52ec25863dd8\") " pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:32:57.966856 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966697 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-system-cni-dir\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:57.966856 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966724 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-multus-socket-dir-parent\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:57.966856 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966745 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-run-systemd\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.966856 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966760 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brwl5\" (UniqueName: \"kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5\") pod \"network-check-target-fvsts\" (UID: \"9602778d-6209-402b-8ccf-4267f9a37ad5\") " pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:32:57.966856 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966777 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca43f479-c01f-49e4-b1fe-a36149b29f51-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:57.966856 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966828 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f61fa573-7943-469d-8262-523ad3aabb94-registration-dir\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:57.967407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966869 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5eb6836b-6d35-49c3-899b-d9e81847d4fb-konnectivity-ca\") pod \"konnectivity-agent-6vrvs\" (UID: \"5eb6836b-6d35-49c3-899b-d9e81847d4fb\") " pod="kube-system/konnectivity-agent-6vrvs" Jul 09 22:32:57.967407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966898 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-os-release\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:57.967407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966922 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-kubelet\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.967407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966947 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpg2w\" (UniqueName: \"kubernetes.io/projected/967a3140-f9ca-4ae6-a61d-ba6b9824796a-kube-api-access-rpg2w\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.967407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966972 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f61fa573-7943-469d-8262-523ad3aabb94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:57.967407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.966995 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-multus-cni-dir\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:57.967407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967021 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-hostroot\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:57.967407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967046 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-run-netns\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.967407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967068 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-node-log\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.967407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967106 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/967a3140-f9ca-4ae6-a61d-ba6b9824796a-ovnkube-script-lib\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.967407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967126 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-host-var-lib-kubelet\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:57.967407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967143 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/967a3140-f9ca-4ae6-a61d-ba6b9824796a-ovnkube-config\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.967407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967162 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-etc-modprobe-d\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:57.967407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967188 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-cnibin\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:57.967407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967232 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca43f479-c01f-49e4-b1fe-a36149b29f51-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:57.967407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967297 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcvsg\" (UniqueName: \"kubernetes.io/projected/f61fa573-7943-469d-8262-523ad3aabb94-kube-api-access-lcvsg\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:57.967407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967325 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5eb6836b-6d35-49c3-899b-d9e81847d4fb-agent-certs\") pod \"konnectivity-agent-6vrvs\" (UID: \"5eb6836b-6d35-49c3-899b-d9e81847d4fb\") " pod="kube-system/konnectivity-agent-6vrvs" Jul 09 22:32:57.968186 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967347 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-host-var-lib-cni-bin\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:57.968186 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967370 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c7df5c30-4311-4d4e-ae82-e8822df20edb-multus-daemon-config\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:57.968186 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967394 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-slash\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.968186 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967428 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-cni-netd\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.968186 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967491 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.968186 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967519 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-etc-sysctl-conf\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:57.968186 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967576 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-var-lib-kubelet\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:57.968186 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967646 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5eb6836b-6d35-49c3-899b-d9e81847d4fb-konnectivity-ca\") pod \"konnectivity-agent-6vrvs\" (UID: \"5eb6836b-6d35-49c3-899b-d9e81847d4fb\") " pod="kube-system/konnectivity-agent-6vrvs" Jul 09 22:32:57.968186 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967651 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8d3fdbc-afb4-486f-a376-b4e70e32bdd8-host\") pod \"node-ca-vsrl6\" (UID: \"b8d3fdbc-afb4-486f-a376-b4e70e32bdd8\") " pod="openshift-image-registry/node-ca-vsrl6" Jul 09 22:32:57.968186 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967699 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-host-run-netns\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:57.968186 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967696 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 22:32:57.968186 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967708 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8d3fdbc-afb4-486f-a376-b4e70e32bdd8-host\") pod \"node-ca-vsrl6\" (UID: \"b8d3fdbc-afb4-486f-a376-b4e70e32bdd8\") " pod="openshift-image-registry/node-ca-vsrl6" Jul 09 22:32:57.968186 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967755 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-run\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:57.968186 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967777 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-sys\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:57.968186 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967824 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vmcm\" (UniqueName: \"kubernetes.io/projected/85d0d04e-7886-4803-82a9-980ae8b0d714-kube-api-access-7vmcm\") pod \"node-resolver-sgpbm\" (UID: \"85d0d04e-7886-4803-82a9-980ae8b0d714\") " pod="openshift-dns/node-resolver-sgpbm" Jul 09 22:32:57.968186 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967851 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f61fa573-7943-469d-8262-523ad3aabb94-sys-fs\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:57.968186 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967873 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-multus-conf-dir\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:57.968980 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967905 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-run-openvswitch\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.968980 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967930 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-etc-kubernetes\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:57.968980 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.967979 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/44657019-d54e-4ff7-8677-c4ec3d34e1f8-dbus\") pod \"global-pull-secret-syncer-9nmjj\" (UID: \"44657019-d54e-4ff7-8677-c4ec3d34e1f8\") " pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:32:57.968980 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968001 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-9nmjj\" (UID: \"44657019-d54e-4ff7-8677-c4ec3d34e1f8\") " pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:32:57.968980 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968018 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-var-lib-openvswitch\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.968980 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968034 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-etc-openvswitch\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.968980 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968055 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-log-socket\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.968980 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968081 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/967a3140-f9ca-4ae6-a61d-ba6b9824796a-env-overrides\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.968980 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968096 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-etc-sysctl-d\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:57.968980 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968110 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/44657019-d54e-4ff7-8677-c4ec3d34e1f8-kubelet-config\") pod \"global-pull-secret-syncer-9nmjj\" (UID: \"44657019-d54e-4ff7-8677-c4ec3d34e1f8\") " pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:32:57.968980 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968128 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-host-run-k8s-cni-cncf-io\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:57.968980 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968164 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-etc-kubernetes\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:57.968980 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968207 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca43f479-c01f-49e4-b1fe-a36149b29f51-system-cni-dir\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:57.968980 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968243 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpwrl\" (UniqueName: \"kubernetes.io/projected/ca43f479-c01f-49e4-b1fe-a36149b29f51-kube-api-access-zpwrl\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:57.968980 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968271 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44c0e227-9c8f-4de3-8923-de936db6502c-etc-tuned\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:57.968980 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968294 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/85d0d04e-7886-4803-82a9-980ae8b0d714-hosts-file\") pod \"node-resolver-sgpbm\" (UID: \"85d0d04e-7886-4803-82a9-980ae8b0d714\") " pod="openshift-dns/node-resolver-sgpbm" Jul 09 22:32:57.969700 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968324 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c341c17b-674a-4de7-a6a3-a428f3a655e3-iptables-alerter-script\") pod \"iptables-alerter-pwnxw\" (UID: \"c341c17b-674a-4de7-a6a3-a428f3a655e3\") " pod="openshift-network-operator/iptables-alerter-pwnxw" Jul 09 22:32:57.969700 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968351 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frxwl\" (UniqueName: \"kubernetes.io/projected/c341c17b-674a-4de7-a6a3-a428f3a655e3-kube-api-access-frxwl\") pod \"iptables-alerter-pwnxw\" (UID: \"c341c17b-674a-4de7-a6a3-a428f3a655e3\") " pod="openshift-network-operator/iptables-alerter-pwnxw" Jul 09 22:32:57.969700 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968378 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-host-var-lib-cni-multus\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:57.969700 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968402 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-systemd-units\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.969700 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968439 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/967a3140-f9ca-4ae6-a61d-ba6b9824796a-ovn-node-metrics-cert\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.969700 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968606 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f61fa573-7943-469d-8262-523ad3aabb94-socket-dir\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:57.969700 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968640 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7df5c30-4311-4d4e-ae82-e8822df20edb-cni-binary-copy\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:57.969700 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968699 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-cni-bin\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:57.969700 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968732 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ca43f479-c01f-49e4-b1fe-a36149b29f51-os-release\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:57.969700 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968760 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-etc-sysconfig\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:57.969700 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968792 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-etc-systemd\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:57.969700 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.968895 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c341c17b-674a-4de7-a6a3-a428f3a655e3-iptables-alerter-script\") pod \"iptables-alerter-pwnxw\" (UID: \"c341c17b-674a-4de7-a6a3-a428f3a655e3\") " pod="openshift-network-operator/iptables-alerter-pwnxw" Jul 09 22:32:57.972699 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.972671 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5eb6836b-6d35-49c3-899b-d9e81847d4fb-agent-certs\") pod \"konnectivity-agent-6vrvs\" (UID: \"5eb6836b-6d35-49c3-899b-d9e81847d4fb\") " pod="kube-system/konnectivity-agent-6vrvs" Jul 09 22:32:57.976439 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.976405 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsx7h\" (UniqueName: \"kubernetes.io/projected/b8d3fdbc-afb4-486f-a376-b4e70e32bdd8-kube-api-access-hsx7h\") pod \"node-ca-vsrl6\" (UID: \"b8d3fdbc-afb4-486f-a376-b4e70e32bdd8\") " pod="openshift-image-registry/node-ca-vsrl6" Jul 09 22:32:57.977202 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:57.977177 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frxwl\" (UniqueName: \"kubernetes.io/projected/c341c17b-674a-4de7-a6a3-a428f3a655e3-kube-api-access-frxwl\") pod \"iptables-alerter-pwnxw\" (UID: \"c341c17b-674a-4de7-a6a3-a428f3a655e3\") " pod="openshift-network-operator/iptables-alerter-pwnxw" Jul 09 22:32:58.047593 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.047531 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" event={"ID":"4bd33866a0efa68864d83a51f0f716fd","Type":"ContainerStarted","Data":"6924dae4f00c111597071421f3e75f40fdd28a086a0a289d610b2abfb052306d"} Jul 09 22:32:58.048812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.048784 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" event={"ID":"acfe4b356a98be4a5be8184b12458310","Type":"ContainerStarted","Data":"386d9f4fb0d2b3b1ae9da5db74be940a7b286c862cf39eefe5e7fb06dc81a494"} Jul 09 22:32:58.069624 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.069592 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws69l\" (UniqueName: \"kubernetes.io/projected/44c0e227-9c8f-4de3-8923-de936db6502c-kube-api-access-ws69l\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.069624 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.069631 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs\") pod \"network-metrics-daemon-5wch9\" (UID: \"941cb948-d902-4c71-abf2-52ec25863dd8\") " pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:32:58.069888 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.069672 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vgdx\" (UniqueName: \"kubernetes.io/projected/c7df5c30-4311-4d4e-ae82-e8822df20edb-kube-api-access-6vgdx\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.069888 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.069697 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-host\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.069888 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.069723 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89zg5\" (UniqueName: \"kubernetes.io/projected/941cb948-d902-4c71-abf2-52ec25863dd8-kube-api-access-89zg5\") pod \"network-metrics-daemon-5wch9\" (UID: \"941cb948-d902-4c71-abf2-52ec25863dd8\") " pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:32:58.069888 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.069745 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-system-cni-dir\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.069888 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:58.069820 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:32:58.070155 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:58.069907 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs podName:941cb948-d902-4c71-abf2-52ec25863dd8 nodeName:}" failed. No retries permitted until 2026-07-09 22:32:58.569879417 +0000 UTC m=+3.110958209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs") pod "network-metrics-daemon-5wch9" (UID: "941cb948-d902-4c71-abf2-52ec25863dd8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:32:58.070155 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.069823 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-system-cni-dir\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.070155 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.069992 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-multus-socket-dir-parent\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.070155 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070000 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-host\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.070155 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070023 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-run-systemd\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.070155 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070050 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brwl5\" (UniqueName: \"kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5\") pod \"network-check-target-fvsts\" (UID: \"9602778d-6209-402b-8ccf-4267f9a37ad5\") " pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:32:58.070155 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070064 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-run-systemd\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.070155 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070076 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca43f479-c01f-49e4-b1fe-a36149b29f51-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:58.070155 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070104 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f61fa573-7943-469d-8262-523ad3aabb94-registration-dir\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:58.070155 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070146 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-os-release\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.070155 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070152 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-multus-socket-dir-parent\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.070747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070171 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-kubelet\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.070747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070198 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpg2w\" (UniqueName: \"kubernetes.io/projected/967a3140-f9ca-4ae6-a61d-ba6b9824796a-kube-api-access-rpg2w\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.070747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070225 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f61fa573-7943-469d-8262-523ad3aabb94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:58.070747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070233 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-os-release\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.070747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070239 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-kubelet\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.070747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070251 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-multus-cni-dir\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.070747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070256 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca43f479-c01f-49e4-b1fe-a36149b29f51-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:58.070747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070232 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f61fa573-7943-469d-8262-523ad3aabb94-registration-dir\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:58.070747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070276 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-hostroot\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.070747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070296 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f61fa573-7943-469d-8262-523ad3aabb94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:58.070747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070320 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-run-netns\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.070747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070327 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-hostroot\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.070747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070343 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-multus-cni-dir\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.070747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070347 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-node-log\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.070747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070381 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-node-log\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.070747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070400 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/967a3140-f9ca-4ae6-a61d-ba6b9824796a-ovnkube-script-lib\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.070747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070430 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-run-netns\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.070747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070446 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-host-var-lib-kubelet\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.071723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070481 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/967a3140-f9ca-4ae6-a61d-ba6b9824796a-ovnkube-config\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.071723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070524 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-etc-modprobe-d\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.071723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070550 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-cnibin\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.071723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070482 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-host-var-lib-kubelet\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.071723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070578 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca43f479-c01f-49e4-b1fe-a36149b29f51-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:58.071723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070607 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcvsg\" (UniqueName: \"kubernetes.io/projected/f61fa573-7943-469d-8262-523ad3aabb94-kube-api-access-lcvsg\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:58.071723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070639 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-host-var-lib-cni-bin\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.071723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070680 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-etc-modprobe-d\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.071723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070695 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c7df5c30-4311-4d4e-ae82-e8822df20edb-multus-daemon-config\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.071723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070729 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-slash\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.071723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070759 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-cni-netd\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.071723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070794 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.071723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070828 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-etc-sysctl-conf\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.071723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070863 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-var-lib-kubelet\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.071723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070896 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-host-run-netns\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.071723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070928 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-run\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.071723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070957 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-sys\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.072608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.070989 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vmcm\" (UniqueName: \"kubernetes.io/projected/85d0d04e-7886-4803-82a9-980ae8b0d714-kube-api-access-7vmcm\") pod \"node-resolver-sgpbm\" (UID: \"85d0d04e-7886-4803-82a9-980ae8b0d714\") " pod="openshift-dns/node-resolver-sgpbm" Jul 09 22:32:58.072608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071025 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f61fa573-7943-469d-8262-523ad3aabb94-sys-fs\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:58.072608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071036 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-host-var-lib-cni-bin\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.072608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071057 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-multus-conf-dir\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.072608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071086 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-run-openvswitch\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.072608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071110 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-etc-kubernetes\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.072608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071113 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-cnibin\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.072608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071129 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/967a3140-f9ca-4ae6-a61d-ba6b9824796a-ovnkube-script-lib\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.072608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071171 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-run\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.072608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071216 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/44657019-d54e-4ff7-8677-c4ec3d34e1f8-dbus\") pod \"global-pull-secret-syncer-9nmjj\" (UID: \"44657019-d54e-4ff7-8677-c4ec3d34e1f8\") " pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:32:58.072608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071227 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-sys\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.072608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071137 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/44657019-d54e-4ff7-8677-c4ec3d34e1f8-dbus\") pod \"global-pull-secret-syncer-9nmjj\" (UID: \"44657019-d54e-4ff7-8677-c4ec3d34e1f8\") " pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:32:58.072608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071265 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.072608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071266 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-9nmjj\" (UID: \"44657019-d54e-4ff7-8677-c4ec3d34e1f8\") " pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:32:58.072608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071300 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-var-lib-openvswitch\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.072608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071325 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-etc-openvswitch\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.072608 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:58.071333 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 22:32:58.072608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071349 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-etc-sysctl-conf\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.073508 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071367 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-log-socket\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.073508 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071381 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca43f479-c01f-49e4-b1fe-a36149b29f51-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:58.073508 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071218 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-slash\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.073508 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071403 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-var-lib-kubelet\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.073508 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:58.071386 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret podName:44657019-d54e-4ff7-8677-c4ec3d34e1f8 nodeName:}" failed. No retries permitted until 2026-07-09 22:32:58.57136805 +0000 UTC m=+3.112446824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret") pod "global-pull-secret-syncer-9nmjj" (UID: "44657019-d54e-4ff7-8677-c4ec3d34e1f8") : object "kube-system"/"original-pull-secret" not registered Jul 09 22:32:58.073508 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071439 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-log-socket\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.073508 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071450 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/967a3140-f9ca-4ae6-a61d-ba6b9824796a-env-overrides\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.073508 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071447 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-host-run-netns\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.073508 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071480 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-etc-sysctl-d\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.073508 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071507 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/44657019-d54e-4ff7-8677-c4ec3d34e1f8-kubelet-config\") pod \"global-pull-secret-syncer-9nmjj\" (UID: \"44657019-d54e-4ff7-8677-c4ec3d34e1f8\") " pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:32:58.073508 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071512 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-etc-kubernetes\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.073508 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071510 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c7df5c30-4311-4d4e-ae82-e8822df20edb-multus-daemon-config\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.073508 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071513 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-run-openvswitch\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.073508 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071545 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-host-run-k8s-cni-cncf-io\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.073508 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071580 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/44657019-d54e-4ff7-8677-c4ec3d34e1f8-kubelet-config\") pod \"global-pull-secret-syncer-9nmjj\" (UID: \"44657019-d54e-4ff7-8677-c4ec3d34e1f8\") " pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:32:58.073508 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071582 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-etc-openvswitch\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.073508 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071588 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-cni-netd\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.074330 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071595 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-etc-kubernetes\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.074330 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071612 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/967a3140-f9ca-4ae6-a61d-ba6b9824796a-ovnkube-config\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.074330 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071634 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-etc-kubernetes\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.074330 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071616 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-host-run-k8s-cni-cncf-io\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.074330 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071642 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f61fa573-7943-469d-8262-523ad3aabb94-sys-fs\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:58.074330 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071655 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-var-lib-openvswitch\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.074330 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071669 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca43f479-c01f-49e4-b1fe-a36149b29f51-system-cni-dir\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:58.074330 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071687 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-multus-conf-dir\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.074330 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071694 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpwrl\" (UniqueName: \"kubernetes.io/projected/ca43f479-c01f-49e4-b1fe-a36149b29f51-kube-api-access-zpwrl\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:58.074330 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071718 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca43f479-c01f-49e4-b1fe-a36149b29f51-system-cni-dir\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:58.074330 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071719 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-etc-sysctl-d\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.074330 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071720 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44c0e227-9c8f-4de3-8923-de936db6502c-etc-tuned\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.074330 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071744 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/85d0d04e-7886-4803-82a9-980ae8b0d714-hosts-file\") pod \"node-resolver-sgpbm\" (UID: \"85d0d04e-7886-4803-82a9-980ae8b0d714\") " pod="openshift-dns/node-resolver-sgpbm" Jul 09 22:32:58.074330 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071770 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-host-var-lib-cni-multus\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.074330 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071787 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-systemd-units\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.074330 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071803 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/967a3140-f9ca-4ae6-a61d-ba6b9824796a-ovn-node-metrics-cert\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.074330 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071819 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f61fa573-7943-469d-8262-523ad3aabb94-socket-dir\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:58.075120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071834 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7df5c30-4311-4d4e-ae82-e8822df20edb-cni-binary-copy\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.075120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071843 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-systemd-units\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.075120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071856 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-cni-bin\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.075120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071882 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ca43f479-c01f-49e4-b1fe-a36149b29f51-os-release\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:58.075120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071885 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-host-var-lib-cni-multus\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.075120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071886 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/85d0d04e-7886-4803-82a9-980ae8b0d714-hosts-file\") pod \"node-resolver-sgpbm\" (UID: \"85d0d04e-7886-4803-82a9-980ae8b0d714\") " pod="openshift-dns/node-resolver-sgpbm" Jul 09 22:32:58.075120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071897 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-etc-sysconfig\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.075120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071933 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-etc-systemd\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.075120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071957 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-run-ovn-kubernetes\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.075120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071963 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/967a3140-f9ca-4ae6-a61d-ba6b9824796a-env-overrides\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.075120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.071979 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca43f479-c01f-49e4-b1fe-a36149b29f51-cnibin\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:58.075120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072016 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca43f479-c01f-49e4-b1fe-a36149b29f51-cnibin\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:58.075120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072040 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f61fa573-7943-469d-8262-523ad3aabb94-device-dir\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:58.075120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072045 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-cni-bin\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.075120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072036 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f61fa573-7943-469d-8262-523ad3aabb94-socket-dir\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:58.075120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072072 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca43f479-c01f-49e4-b1fe-a36149b29f51-cni-binary-copy\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:58.075120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072078 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-etc-sysconfig\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.075919 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072130 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-host-run-ovn-kubernetes\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.075919 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072140 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ca43f479-c01f-49e4-b1fe-a36149b29f51-os-release\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:58.075919 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072173 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-etc-systemd\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.075919 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072180 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca43f479-c01f-49e4-b1fe-a36149b29f51-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:58.075919 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072188 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f61fa573-7943-469d-8262-523ad3aabb94-device-dir\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:58.075919 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072223 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-lib-modules\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.075919 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072260 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85d0d04e-7886-4803-82a9-980ae8b0d714-tmp-dir\") pod \"node-resolver-sgpbm\" (UID: \"85d0d04e-7886-4803-82a9-980ae8b0d714\") " pod="openshift-dns/node-resolver-sgpbm" Jul 09 22:32:58.075919 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072284 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f61fa573-7943-469d-8262-523ad3aabb94-etc-selinux\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:58.075919 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072310 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44c0e227-9c8f-4de3-8923-de936db6502c-lib-modules\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.075919 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072313 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-host-run-multus-certs\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.075919 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072348 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c7df5c30-4311-4d4e-ae82-e8822df20edb-host-run-multus-certs\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.075919 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072352 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-run-ovn\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.075919 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072381 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44c0e227-9c8f-4de3-8923-de936db6502c-tmp\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.075919 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072387 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/967a3140-f9ca-4ae6-a61d-ba6b9824796a-run-ovn\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.075919 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.072911 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f61fa573-7943-469d-8262-523ad3aabb94-etc-selinux\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:58.075919 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.073292 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca43f479-c01f-49e4-b1fe-a36149b29f51-cni-binary-copy\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:58.075919 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.073387 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca43f479-c01f-49e4-b1fe-a36149b29f51-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:58.076695 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.073744 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7df5c30-4311-4d4e-ae82-e8822df20edb-cni-binary-copy\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.076695 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.074609 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/967a3140-f9ca-4ae6-a61d-ba6b9824796a-ovn-node-metrics-cert\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.076695 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.074613 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/44c0e227-9c8f-4de3-8923-de936db6502c-etc-tuned\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.077355 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.077332 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44c0e227-9c8f-4de3-8923-de936db6502c-tmp\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.077476 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.077454 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85d0d04e-7886-4803-82a9-980ae8b0d714-tmp-dir\") pod \"node-resolver-sgpbm\" (UID: \"85d0d04e-7886-4803-82a9-980ae8b0d714\") " pod="openshift-dns/node-resolver-sgpbm" Jul 09 22:32:58.078070 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:58.077907 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 22:32:58.078070 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:58.077932 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 22:32:58.078070 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:58.077948 2568 projected.go:196] Error preparing data for projected volume kube-api-access-brwl5 for pod openshift-network-diagnostics/network-check-target-fvsts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:32:58.078070 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:58.078024 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5 podName:9602778d-6209-402b-8ccf-4267f9a37ad5 nodeName:}" failed. No retries permitted until 2026-07-09 22:32:58.578005257 +0000 UTC m=+3.119084053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-brwl5" (UniqueName: "kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5") pod "network-check-target-fvsts" (UID: "9602778d-6209-402b-8ccf-4267f9a37ad5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:32:58.080732 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.080706 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpg2w\" (UniqueName: \"kubernetes.io/projected/967a3140-f9ca-4ae6-a61d-ba6b9824796a-kube-api-access-rpg2w\") pod \"ovnkube-node-pm8p2\" (UID: \"967a3140-f9ca-4ae6-a61d-ba6b9824796a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.080857 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.080711 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws69l\" (UniqueName: \"kubernetes.io/projected/44c0e227-9c8f-4de3-8923-de936db6502c-kube-api-access-ws69l\") pod \"tuned-8nxpc\" (UID: \"44c0e227-9c8f-4de3-8923-de936db6502c\") " pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.081118 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.081069 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89zg5\" (UniqueName: \"kubernetes.io/projected/941cb948-d902-4c71-abf2-52ec25863dd8-kube-api-access-89zg5\") pod \"network-metrics-daemon-5wch9\" (UID: \"941cb948-d902-4c71-abf2-52ec25863dd8\") " pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:32:58.081236 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.081154 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vgdx\" (UniqueName: \"kubernetes.io/projected/c7df5c30-4311-4d4e-ae82-e8822df20edb-kube-api-access-6vgdx\") pod \"multus-c8lxb\" (UID: \"c7df5c30-4311-4d4e-ae82-e8822df20edb\") " pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.081601 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.081579 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpwrl\" (UniqueName: \"kubernetes.io/projected/ca43f479-c01f-49e4-b1fe-a36149b29f51-kube-api-access-zpwrl\") pod \"multus-additional-cni-plugins-8ddvw\" (UID: \"ca43f479-c01f-49e4-b1fe-a36149b29f51\") " pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:58.082142 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.082118 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcvsg\" (UniqueName: \"kubernetes.io/projected/f61fa573-7943-469d-8262-523ad3aabb94-kube-api-access-lcvsg\") pod \"aws-ebs-csi-driver-node-6wbr2\" (UID: \"f61fa573-7943-469d-8262-523ad3aabb94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:58.082288 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.082265 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vmcm\" (UniqueName: \"kubernetes.io/projected/85d0d04e-7886-4803-82a9-980ae8b0d714-kube-api-access-7vmcm\") pod \"node-resolver-sgpbm\" (UID: \"85d0d04e-7886-4803-82a9-980ae8b0d714\") " pod="openshift-dns/node-resolver-sgpbm" Jul 09 22:32:58.164523 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.164362 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pwnxw" Jul 09 22:32:58.171424 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.171386 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vsrl6" Jul 09 22:32:58.181176 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.181142 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6vrvs" Jul 09 22:32:58.186496 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:58.186455 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc341c17b_674a_4de7_a6a3_a428f3a655e3.slice/crio-7161da330b8def10226feb358a9781c6c8784c052f61605e8c32bde8d08d3aeb WatchSource:0}: Error finding container 7161da330b8def10226feb358a9781c6c8784c052f61605e8c32bde8d08d3aeb: Status 404 returned error can't find the container with id 7161da330b8def10226feb358a9781c6c8784c052f61605e8c32bde8d08d3aeb Jul 09 22:32:58.186815 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:58.186791 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d3fdbc_afb4_486f_a376_b4e70e32bdd8.slice/crio-48b43f1b034345d247cf1459a3d8b49f7394d44ff6879ff56f0834b71d3f2875 WatchSource:0}: Error finding container 48b43f1b034345d247cf1459a3d8b49f7394d44ff6879ff56f0834b71d3f2875: Status 404 returned error can't find the container with id 48b43f1b034345d247cf1459a3d8b49f7394d44ff6879ff56f0834b71d3f2875 Jul 09 22:32:58.188944 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.188930 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" Jul 09 22:32:58.198189 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.198159 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8ddvw" Jul 09 22:32:58.199966 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:58.199936 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb6836b_6d35_49c3_899b_d9e81847d4fb.slice/crio-f2b9d7a9de1e6a7e4c601eff2d2791162be9caa53d3d920ef982c1ab47d254a1 WatchSource:0}: Error finding container f2b9d7a9de1e6a7e4c601eff2d2791162be9caa53d3d920ef982c1ab47d254a1: Status 404 returned error can't find the container with id f2b9d7a9de1e6a7e4c601eff2d2791162be9caa53d3d920ef982c1ab47d254a1 Jul 09 22:32:58.204903 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.204878 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c8lxb" Jul 09 22:32:58.208276 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:58.208241 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf61fa573_7943_469d_8262_523ad3aabb94.slice/crio-b096beafba5a26fdca2bad29c957fcb7dcc3aca145d2a99b02169634da418960 WatchSource:0}: Error finding container b096beafba5a26fdca2bad29c957fcb7dcc3aca145d2a99b02169634da418960: Status 404 returned error can't find the container with id b096beafba5a26fdca2bad29c957fcb7dcc3aca145d2a99b02169634da418960 Jul 09 22:32:58.212851 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.212828 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:32:58.221287 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:58.220884 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca43f479_c01f_49e4_b1fe_a36149b29f51.slice/crio-50f917fdc4ac46331fb6d029bd9015c786bf79c4c2d5e814921d9af0a7cb197a WatchSource:0}: Error finding container 50f917fdc4ac46331fb6d029bd9015c786bf79c4c2d5e814921d9af0a7cb197a: Status 404 returned error can't find the container with id 50f917fdc4ac46331fb6d029bd9015c786bf79c4c2d5e814921d9af0a7cb197a Jul 09 22:32:58.221287 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.221043 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" Jul 09 22:32:58.223232 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:58.222980 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7df5c30_4311_4d4e_ae82_e8822df20edb.slice/crio-d5c3dd879f40a2dcfbd5b5a26341e1fc0d37b538a5ddb84d374a1a30550a0b4f WatchSource:0}: Error finding container d5c3dd879f40a2dcfbd5b5a26341e1fc0d37b538a5ddb84d374a1a30550a0b4f: Status 404 returned error can't find the container with id d5c3dd879f40a2dcfbd5b5a26341e1fc0d37b538a5ddb84d374a1a30550a0b4f Jul 09 22:32:58.227822 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.227586 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sgpbm" Jul 09 22:32:58.229298 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:58.229252 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967a3140_f9ca_4ae6_a61d_ba6b9824796a.slice/crio-d177a76cfdd2bcb1df38021106d87b4441421058c01eafe3c05ef9937c551b24 WatchSource:0}: Error finding container d177a76cfdd2bcb1df38021106d87b4441421058c01eafe3c05ef9937c551b24: Status 404 returned error can't find the container with id d177a76cfdd2bcb1df38021106d87b4441421058c01eafe3c05ef9937c551b24 Jul 09 22:32:58.239853 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:58.239819 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c0e227_9c8f_4de3_8923_de936db6502c.slice/crio-472f3d2ecabe0245334b62e7353c4f87aed22be64065b8c97b08ffde8a3fc172 WatchSource:0}: Error finding container 472f3d2ecabe0245334b62e7353c4f87aed22be64065b8c97b08ffde8a3fc172: Status 404 returned error can't find the container with id 472f3d2ecabe0245334b62e7353c4f87aed22be64065b8c97b08ffde8a3fc172 Jul 09 22:32:58.244820 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:32:58.244756 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d0d04e_7886_4803_82a9_980ae8b0d714.slice/crio-743ca57ecb7b6db49a1969f17bf1da082d1a4a025b22af8375b0a49057092208 WatchSource:0}: Error finding container 743ca57ecb7b6db49a1969f17bf1da082d1a4a025b22af8375b0a49057092208: Status 404 returned error can't find the container with id 743ca57ecb7b6db49a1969f17bf1da082d1a4a025b22af8375b0a49057092208 Jul 09 22:32:58.315081 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.315045 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 22:32:58.578686 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.578592 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs\") pod \"network-metrics-daemon-5wch9\" (UID: \"941cb948-d902-4c71-abf2-52ec25863dd8\") " pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:32:58.578686 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.578637 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brwl5\" (UniqueName: \"kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5\") pod \"network-check-target-fvsts\" (UID: \"9602778d-6209-402b-8ccf-4267f9a37ad5\") " pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:32:58.578935 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.578695 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-9nmjj\" (UID: \"44657019-d54e-4ff7-8677-c4ec3d34e1f8\") " pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:32:58.578935 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:58.578824 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 22:32:58.578935 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:58.578891 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret podName:44657019-d54e-4ff7-8677-c4ec3d34e1f8 nodeName:}" failed. No retries permitted until 2026-07-09 22:32:59.578871388 +0000 UTC m=+4.119950162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret") pod "global-pull-secret-syncer-9nmjj" (UID: "44657019-d54e-4ff7-8677-c4ec3d34e1f8") : object "kube-system"/"original-pull-secret" not registered Jul 09 22:32:58.579271 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:58.579218 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 22:32:58.579271 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:58.579230 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:32:58.579271 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:58.579245 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 22:32:58.579271 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:58.579262 2568 projected.go:196] Error preparing data for projected volume kube-api-access-brwl5 for pod openshift-network-diagnostics/network-check-target-fvsts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:32:58.579575 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:58.579298 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs podName:941cb948-d902-4c71-abf2-52ec25863dd8 nodeName:}" failed. No retries permitted until 2026-07-09 22:32:59.579279716 +0000 UTC m=+4.120358498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs") pod "network-metrics-daemon-5wch9" (UID: "941cb948-d902-4c71-abf2-52ec25863dd8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:32:58.579575 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:58.579322 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5 podName:9602778d-6209-402b-8ccf-4267f9a37ad5 nodeName:}" failed. No retries permitted until 2026-07-09 22:32:59.579310771 +0000 UTC m=+4.120389545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-brwl5" (UniqueName: "kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5") pod "network-check-target-fvsts" (UID: "9602778d-6209-402b-8ccf-4267f9a37ad5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:32:58.895098 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.894961 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 22:27:57 +0000 UTC" deadline="2028-04-14 04:39:59.37353928 +0000 UTC" Jul 09 22:32:58.895098 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:58.895047 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15462h7m0.478496487s" Jul 09 22:32:59.051609 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:59.051569 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" event={"ID":"44c0e227-9c8f-4de3-8923-de936db6502c","Type":"ContainerStarted","Data":"472f3d2ecabe0245334b62e7353c4f87aed22be64065b8c97b08ffde8a3fc172"} Jul 09 22:32:59.052829 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:59.052797 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c8lxb" event={"ID":"c7df5c30-4311-4d4e-ae82-e8822df20edb","Type":"ContainerStarted","Data":"d5c3dd879f40a2dcfbd5b5a26341e1fc0d37b538a5ddb84d374a1a30550a0b4f"} Jul 09 22:32:59.053963 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:59.053938 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ddvw" event={"ID":"ca43f479-c01f-49e4-b1fe-a36149b29f51","Type":"ContainerStarted","Data":"50f917fdc4ac46331fb6d029bd9015c786bf79c4c2d5e814921d9af0a7cb197a"} Jul 09 22:32:59.055092 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:59.055064 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6vrvs" event={"ID":"5eb6836b-6d35-49c3-899b-d9e81847d4fb","Type":"ContainerStarted","Data":"f2b9d7a9de1e6a7e4c601eff2d2791162be9caa53d3d920ef982c1ab47d254a1"} Jul 09 22:32:59.056299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:59.056224 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pwnxw" event={"ID":"c341c17b-674a-4de7-a6a3-a428f3a655e3","Type":"ContainerStarted","Data":"7161da330b8def10226feb358a9781c6c8784c052f61605e8c32bde8d08d3aeb"} Jul 09 22:32:59.057348 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:59.057323 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sgpbm" event={"ID":"85d0d04e-7886-4803-82a9-980ae8b0d714","Type":"ContainerStarted","Data":"743ca57ecb7b6db49a1969f17bf1da082d1a4a025b22af8375b0a49057092208"} Jul 09 22:32:59.058534 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:59.058513 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" event={"ID":"967a3140-f9ca-4ae6-a61d-ba6b9824796a","Type":"ContainerStarted","Data":"d177a76cfdd2bcb1df38021106d87b4441421058c01eafe3c05ef9937c551b24"} Jul 09 22:32:59.059651 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:59.059624 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" event={"ID":"f61fa573-7943-469d-8262-523ad3aabb94","Type":"ContainerStarted","Data":"b096beafba5a26fdca2bad29c957fcb7dcc3aca145d2a99b02169634da418960"} Jul 09 22:32:59.060761 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:59.060739 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vsrl6" event={"ID":"b8d3fdbc-afb4-486f-a376-b4e70e32bdd8","Type":"ContainerStarted","Data":"48b43f1b034345d247cf1459a3d8b49f7394d44ff6879ff56f0834b71d3f2875"} Jul 09 22:32:59.586916 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:59.586875 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-9nmjj\" (UID: \"44657019-d54e-4ff7-8677-c4ec3d34e1f8\") " pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:32:59.587094 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:59.586964 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs\") pod \"network-metrics-daemon-5wch9\" (UID: \"941cb948-d902-4c71-abf2-52ec25863dd8\") " pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:32:59.587094 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:59.586996 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brwl5\" (UniqueName: \"kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5\") pod \"network-check-target-fvsts\" (UID: \"9602778d-6209-402b-8ccf-4267f9a37ad5\") " pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:32:59.587214 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:59.587151 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 22:32:59.587214 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:59.587171 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 22:32:59.587214 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:59.587196 2568 projected.go:196] Error preparing data for projected volume kube-api-access-brwl5 for pod openshift-network-diagnostics/network-check-target-fvsts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:32:59.587367 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:59.587258 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5 podName:9602778d-6209-402b-8ccf-4267f9a37ad5 nodeName:}" failed. No retries permitted until 2026-07-09 22:33:01.587239355 +0000 UTC m=+6.128318149 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-brwl5" (UniqueName: "kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5") pod "network-check-target-fvsts" (UID: "9602778d-6209-402b-8ccf-4267f9a37ad5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:32:59.587697 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:59.587676 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 22:32:59.587790 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:59.587735 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret podName:44657019-d54e-4ff7-8677-c4ec3d34e1f8 nodeName:}" failed. No retries permitted until 2026-07-09 22:33:01.587720922 +0000 UTC m=+6.128799701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret") pod "global-pull-secret-syncer-9nmjj" (UID: "44657019-d54e-4ff7-8677-c4ec3d34e1f8") : object "kube-system"/"original-pull-secret" not registered Jul 09 22:32:59.587921 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:59.587793 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:32:59.587921 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:32:59.587823 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs podName:941cb948-d902-4c71-abf2-52ec25863dd8 nodeName:}" failed. No retries permitted until 2026-07-09 22:33:01.58781284 +0000 UTC m=+6.128891615 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs") pod "network-metrics-daemon-5wch9" (UID: "941cb948-d902-4c71-abf2-52ec25863dd8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:32:59.896459 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:59.896195 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 22:27:57 +0000 UTC" deadline="2028-01-23 11:28:07.044258477 +0000 UTC" Jul 09 22:32:59.896459 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:32:59.896453 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13500h55m7.147812623s" Jul 09 22:33:00.044603 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:00.044480 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:00.044754 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:00.044640 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:33:00.045056 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:00.045037 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:00.045165 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:00.045143 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nmjj" podUID="44657019-d54e-4ff7-8677-c4ec3d34e1f8" Jul 09 22:33:00.045224 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:00.045201 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:00.045278 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:00.045266 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvsts" podUID="9602778d-6209-402b-8ccf-4267f9a37ad5" Jul 09 22:33:00.093063 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:00.093023 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" event={"ID":"4bd33866a0efa68864d83a51f0f716fd","Type":"ContainerStarted","Data":"5fcfd3b94a323a2d61771fec51f37c733837d752762854eed7e4386c47015927"} Jul 09 22:33:01.121906 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:01.121863 2568 generic.go:356] "Generic (PLEG): container finished" podID="acfe4b356a98be4a5be8184b12458310" containerID="10a98064821676da03d8e298b78d8f99657d9ac6963b4948df6fbf64167d01d1" exitCode=0 Jul 09 22:33:01.122476 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:01.122439 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" event={"ID":"acfe4b356a98be4a5be8184b12458310","Type":"ContainerDied","Data":"10a98064821676da03d8e298b78d8f99657d9ac6963b4948df6fbf64167d01d1"} Jul 09 22:33:01.138460 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:01.138323 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-80.ec2.internal" podStartSLOduration=4.138302161 podStartE2EDuration="4.138302161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:33:00.112313327 +0000 UTC m=+4.653392122" watchObservedRunningTime="2026-07-09 22:33:01.138302161 +0000 UTC m=+5.679380959" Jul 09 22:33:01.607558 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:01.607460 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs\") pod \"network-metrics-daemon-5wch9\" (UID: \"941cb948-d902-4c71-abf2-52ec25863dd8\") " pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:01.607558 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:01.607532 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brwl5\" (UniqueName: \"kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5\") pod \"network-check-target-fvsts\" (UID: \"9602778d-6209-402b-8ccf-4267f9a37ad5\") " pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:01.607797 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:01.607583 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-9nmjj\" (UID: \"44657019-d54e-4ff7-8677-c4ec3d34e1f8\") " pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:01.607797 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:01.607702 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 22:33:01.607797 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:01.607767 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret podName:44657019-d54e-4ff7-8677-c4ec3d34e1f8 nodeName:}" failed. No retries permitted until 2026-07-09 22:33:05.607747834 +0000 UTC m=+10.148826610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret") pod "global-pull-secret-syncer-9nmjj" (UID: "44657019-d54e-4ff7-8677-c4ec3d34e1f8") : object "kube-system"/"original-pull-secret" not registered Jul 09 22:33:01.608179 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:01.608157 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:33:01.608273 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:01.608217 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs podName:941cb948-d902-4c71-abf2-52ec25863dd8 nodeName:}" failed. No retries permitted until 2026-07-09 22:33:05.608199448 +0000 UTC m=+10.149278227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs") pod "network-metrics-daemon-5wch9" (UID: "941cb948-d902-4c71-abf2-52ec25863dd8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:33:01.608337 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:01.608294 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 22:33:01.608337 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:01.608308 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 22:33:01.608337 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:01.608320 2568 projected.go:196] Error preparing data for projected volume kube-api-access-brwl5 for pod openshift-network-diagnostics/network-check-target-fvsts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:33:01.608482 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:01.608354 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5 podName:9602778d-6209-402b-8ccf-4267f9a37ad5 nodeName:}" failed. No retries permitted until 2026-07-09 22:33:05.608341753 +0000 UTC m=+10.149420527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-brwl5" (UniqueName: "kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5") pod "network-check-target-fvsts" (UID: "9602778d-6209-402b-8ccf-4267f9a37ad5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:33:02.043620 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:02.042608 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:02.043620 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:02.042751 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvsts" podUID="9602778d-6209-402b-8ccf-4267f9a37ad5" Jul 09 22:33:02.043620 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:02.043212 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:02.043620 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:02.043312 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nmjj" podUID="44657019-d54e-4ff7-8677-c4ec3d34e1f8" Jul 09 22:33:02.043620 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:02.043374 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:02.043620 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:02.043473 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:33:04.042535 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:04.042425 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:04.042535 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:04.042425 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:04.043119 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:04.042571 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:04.043119 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:04.042585 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nmjj" podUID="44657019-d54e-4ff7-8677-c4ec3d34e1f8" Jul 09 22:33:04.043119 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:04.042673 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvsts" podUID="9602778d-6209-402b-8ccf-4267f9a37ad5" Jul 09 22:33:04.043119 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:04.042776 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:33:05.642796 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:05.641789 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brwl5\" (UniqueName: \"kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5\") pod \"network-check-target-fvsts\" (UID: \"9602778d-6209-402b-8ccf-4267f9a37ad5\") " pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:05.642796 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:05.641866 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-9nmjj\" (UID: \"44657019-d54e-4ff7-8677-c4ec3d34e1f8\") " pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:05.642796 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:05.641916 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs\") pod \"network-metrics-daemon-5wch9\" (UID: \"941cb948-d902-4c71-abf2-52ec25863dd8\") " pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:05.642796 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:05.642041 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:33:05.642796 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:05.642107 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs podName:941cb948-d902-4c71-abf2-52ec25863dd8 nodeName:}" failed. No retries permitted until 2026-07-09 22:33:13.642086117 +0000 UTC m=+18.183164894 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs") pod "network-metrics-daemon-5wch9" (UID: "941cb948-d902-4c71-abf2-52ec25863dd8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:33:05.642796 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:05.642578 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 22:33:05.642796 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:05.642598 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 22:33:05.642796 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:05.642612 2568 projected.go:196] Error preparing data for projected volume kube-api-access-brwl5 for pod openshift-network-diagnostics/network-check-target-fvsts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:33:05.642796 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:05.642661 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5 podName:9602778d-6209-402b-8ccf-4267f9a37ad5 nodeName:}" failed. No retries permitted until 2026-07-09 22:33:13.642645063 +0000 UTC m=+18.183723849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-brwl5" (UniqueName: "kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5") pod "network-check-target-fvsts" (UID: "9602778d-6209-402b-8ccf-4267f9a37ad5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:33:05.642796 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:05.642722 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 22:33:05.642796 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:05.642758 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret podName:44657019-d54e-4ff7-8677-c4ec3d34e1f8 nodeName:}" failed. No retries permitted until 2026-07-09 22:33:13.642746259 +0000 UTC m=+18.183825037 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret") pod "global-pull-secret-syncer-9nmjj" (UID: "44657019-d54e-4ff7-8677-c4ec3d34e1f8") : object "kube-system"/"original-pull-secret" not registered Jul 09 22:33:06.043132 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:06.043023 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:06.043311 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:06.043136 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:33:06.043537 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:06.043518 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:06.043650 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:06.043608 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nmjj" podUID="44657019-d54e-4ff7-8677-c4ec3d34e1f8" Jul 09 22:33:06.043709 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:06.043672 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:06.043803 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:06.043780 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvsts" podUID="9602778d-6209-402b-8ccf-4267f9a37ad5" Jul 09 22:33:08.042483 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:08.041982 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:08.042483 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:08.042126 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvsts" podUID="9602778d-6209-402b-8ccf-4267f9a37ad5" Jul 09 22:33:08.042483 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:08.042189 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:08.042483 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:08.042283 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:33:08.042483 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:08.042429 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:08.043160 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:08.042530 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nmjj" podUID="44657019-d54e-4ff7-8677-c4ec3d34e1f8" Jul 09 22:33:10.042439 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:10.042304 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:10.042439 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:10.042304 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:10.042439 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:10.042437 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:10.043057 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:10.042445 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nmjj" podUID="44657019-d54e-4ff7-8677-c4ec3d34e1f8" Jul 09 22:33:10.043057 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:10.042572 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvsts" podUID="9602778d-6209-402b-8ccf-4267f9a37ad5" Jul 09 22:33:10.043057 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:10.042673 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:33:10.144734 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:10.144695 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" event={"ID":"f61fa573-7943-469d-8262-523ad3aabb94","Type":"ContainerStarted","Data":"ac86a10a3a0c1cf0732a5e697cc3df3f75700387d63e73caf79e71a378de8985"} Jul 09 22:33:10.146376 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:10.146335 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vsrl6" event={"ID":"b8d3fdbc-afb4-486f-a376-b4e70e32bdd8","Type":"ContainerStarted","Data":"cd9f9eba0459517cfbd523ca0a8446f890d8c66d2bdc4eb758f2f9be3090e8e1"} Jul 09 22:33:10.149265 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:10.149237 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" event={"ID":"acfe4b356a98be4a5be8184b12458310","Type":"ContainerStarted","Data":"ea4d83e018760ea8251fd47eec6aba46c451de48e8ac7f6911a997400495a616"} Jul 09 22:33:10.151005 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:10.150980 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" event={"ID":"44c0e227-9c8f-4de3-8923-de936db6502c","Type":"ContainerStarted","Data":"4133bd5cc915817bb96c50461252cc952743706f4495a2e14f6a68a83bb9b4bf"} Jul 09 22:33:10.152862 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:10.152834 2568 generic.go:356] "Generic (PLEG): container finished" podID="ca43f479-c01f-49e4-b1fe-a36149b29f51" containerID="628b637ec8557213b24842bec42331ff0b025892cc2865b662a2329444ec053f" exitCode=0 Jul 09 22:33:10.152989 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:10.152913 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ddvw" event={"ID":"ca43f479-c01f-49e4-b1fe-a36149b29f51","Type":"ContainerDied","Data":"628b637ec8557213b24842bec42331ff0b025892cc2865b662a2329444ec053f"} Jul 09 22:33:10.154804 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:10.154673 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6vrvs" event={"ID":"5eb6836b-6d35-49c3-899b-d9e81847d4fb","Type":"ContainerStarted","Data":"9574bc81009f7d972f94db5739c898ff9be6884ecd071876745e81e49fbd793e"} Jul 09 22:33:10.156186 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:10.156162 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sgpbm" event={"ID":"85d0d04e-7886-4803-82a9-980ae8b0d714","Type":"ContainerStarted","Data":"dbe04e5a103a82fc57b94d84973b5f37acbf01a3b96ef076064dd014a7e2d239"} Jul 09 22:33:10.162555 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:10.162506 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vsrl6" podStartSLOduration=2.985636033 podStartE2EDuration="14.162473991s" totalImagesPullingTime="11.176837958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:33:10.161998133 +0000 UTC m=+14.703076949" watchObservedRunningTime="2026-07-09 22:33:10.162473991 +0000 UTC m=+14.703552788" Jul 09 22:33:10.193878 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:10.193820 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8nxpc" podStartSLOduration=3.06892916 podStartE2EDuration="14.193797577s" totalImagesPullingTime="11.124868417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:33:10.178334208 +0000 UTC m=+14.719413015" watchObservedRunningTime="2026-07-09 22:33:10.193797577 +0000 UTC m=+14.734876394" Jul 09 22:33:10.194057 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:10.193950 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sgpbm" podStartSLOduration=3.065802082 podStartE2EDuration="14.193943655s" totalImagesPullingTime="11.128141573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:33:10.193537337 +0000 UTC m=+14.734616134" watchObservedRunningTime="2026-07-09 22:33:10.193943655 +0000 UTC m=+14.735022450" Jul 09 22:33:10.227459 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:10.227407 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-80.ec2.internal" podStartSLOduration=13.227390591 podStartE2EDuration="13.227390591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:33:10.210608384 +0000 UTC m=+14.751687189" watchObservedRunningTime="2026-07-09 22:33:10.227390591 +0000 UTC m=+14.768469384" Jul 09 22:33:10.227708 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:10.227672 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6vrvs" podStartSLOduration=3.060438346 podStartE2EDuration="14.2276621s" totalImagesPullingTime="11.167223754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:33:10.227211761 +0000 UTC m=+14.768290555" watchObservedRunningTime="2026-07-09 22:33:10.2276621 +0000 UTC m=+14.768740894" Jul 09 22:33:10.350285 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:10.350223 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:10.428611 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:10.428520 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:12.041964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:12.041923 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:12.042416 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:12.041932 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:12.042416 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:12.042070 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:33:12.042416 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:12.041942 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:12.042416 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:12.042160 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nmjj" podUID="44657019-d54e-4ff7-8677-c4ec3d34e1f8" Jul 09 22:33:12.042416 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:12.042189 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvsts" podUID="9602778d-6209-402b-8ccf-4267f9a37ad5" Jul 09 22:33:12.162561 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:12.162481 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pwnxw" event={"ID":"c341c17b-674a-4de7-a6a3-a428f3a655e3","Type":"ContainerStarted","Data":"d4a79624526e0381153798b566ff4cf51a7dd86ff63451ca1be78926dc785199"} Jul 09 22:33:12.179119 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:12.179057 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pwnxw" podStartSLOduration=5.000439373 podStartE2EDuration="16.179040647s" totalImagesPullingTime="11.178601274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:33:12.179014191 +0000 UTC m=+16.720093015" watchObservedRunningTime="2026-07-09 22:33:12.179040647 +0000 UTC m=+16.720119499" Jul 09 22:33:12.301230 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:12.301145 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:12.359199 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:12.359160 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:13.301751 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:13.301706 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:13.361630 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:13.361592 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:13.585307 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:13.585275 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:13.645853 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:13.645817 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:13.705407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:13.705371 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs\") pod \"network-metrics-daemon-5wch9\" (UID: \"941cb948-d902-4c71-abf2-52ec25863dd8\") " pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:13.705407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:13.705421 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brwl5\" (UniqueName: \"kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5\") pod \"network-check-target-fvsts\" (UID: \"9602778d-6209-402b-8ccf-4267f9a37ad5\") " pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:13.705687 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:13.705475 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-9nmjj\" (UID: \"44657019-d54e-4ff7-8677-c4ec3d34e1f8\") " pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:13.705687 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:13.705555 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:33:13.705687 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:13.705598 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 22:33:13.705687 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:13.705623 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 22:33:13.705687 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:13.705645 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 22:33:13.705687 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:13.705657 2568 projected.go:196] Error preparing data for projected volume kube-api-access-brwl5 for pod openshift-network-diagnostics/network-check-target-fvsts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:33:13.705996 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:13.705628 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs podName:941cb948-d902-4c71-abf2-52ec25863dd8 nodeName:}" failed. No retries permitted until 2026-07-09 22:33:29.705608491 +0000 UTC m=+34.246687264 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs") pod "network-metrics-daemon-5wch9" (UID: "941cb948-d902-4c71-abf2-52ec25863dd8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:33:13.705996 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:13.705721 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret podName:44657019-d54e-4ff7-8677-c4ec3d34e1f8 nodeName:}" failed. No retries permitted until 2026-07-09 22:33:29.705703073 +0000 UTC m=+34.246781851 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret") pod "global-pull-secret-syncer-9nmjj" (UID: "44657019-d54e-4ff7-8677-c4ec3d34e1f8") : object "kube-system"/"original-pull-secret" not registered Jul 09 22:33:13.705996 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:13.705739 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5 podName:9602778d-6209-402b-8ccf-4267f9a37ad5 nodeName:}" failed. No retries permitted until 2026-07-09 22:33:29.705729745 +0000 UTC m=+34.246808523 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-brwl5" (UniqueName: "kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5") pod "network-check-target-fvsts" (UID: "9602778d-6209-402b-8ccf-4267f9a37ad5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:33:14.042307 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:14.042224 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:14.042478 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:14.042379 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvsts" podUID="9602778d-6209-402b-8ccf-4267f9a37ad5" Jul 09 22:33:14.042478 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:14.042424 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:14.042606 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:14.042559 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:33:14.042606 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:14.042595 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:14.042712 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:14.042682 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nmjj" podUID="44657019-d54e-4ff7-8677-c4ec3d34e1f8" Jul 09 22:33:15.104067 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:15.104012 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6vrvs" Jul 09 22:33:15.104849 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:15.104827 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6vrvs" Jul 09 22:33:16.042891 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:16.042779 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:16.043261 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:16.043168 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:33:16.043261 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:16.043189 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:16.043412 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:16.043294 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nmjj" podUID="44657019-d54e-4ff7-8677-c4ec3d34e1f8" Jul 09 22:33:16.043412 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:16.043324 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:16.043412 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:16.043390 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvsts" podUID="9602778d-6209-402b-8ccf-4267f9a37ad5" Jul 09 22:33:17.208431 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:17.208398 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6vrvs" Jul 09 22:33:17.208898 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:17.208528 2568 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 22:33:17.209430 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:17.209403 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6vrvs" Jul 09 22:33:17.276863 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:17.276816 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:18.045368 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:18.043885 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:18.045368 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:18.044262 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:33:18.045368 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:18.044573 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:18.045368 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:18.044675 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nmjj" podUID="44657019-d54e-4ff7-8677-c4ec3d34e1f8" Jul 09 22:33:18.045368 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:18.044738 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:18.045368 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:18.044805 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvsts" podUID="9602778d-6209-402b-8ccf-4267f9a37ad5" Jul 09 22:33:18.126189 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:18.125590 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 22:33:18.178126 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:18.177941 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c8lxb" event={"ID":"c7df5c30-4311-4d4e-ae82-e8822df20edb","Type":"ContainerStarted","Data":"c475da61d2eaf180903ee0f41e18d5bc9fd910e2d7b55134b269f8e676efd53a"} Jul 09 22:33:18.180121 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:18.180044 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" event={"ID":"967a3140-f9ca-4ae6-a61d-ba6b9824796a","Type":"ContainerStarted","Data":"9c504df931f3ad73501a601ce425b7da06c979e891bdb3c31846add54dd97b11"} Jul 09 22:33:18.180324 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:18.180132 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" event={"ID":"967a3140-f9ca-4ae6-a61d-ba6b9824796a","Type":"ContainerStarted","Data":"4ea9cc69180daf3df297faf0e46950cb0a4b0b021c24918cdb8e49e10c548e34"} Jul 09 22:33:18.182835 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:18.182776 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" event={"ID":"f61fa573-7943-469d-8262-523ad3aabb94","Type":"ContainerStarted","Data":"037dcef2dba3d8338973bcdda8f69ed3ba1c13f53e7d07a6764e0e55d8469464"} Jul 09 22:33:18.197247 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:18.197198 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c8lxb" podStartSLOduration=2.4536586 podStartE2EDuration="22.197173592s" totalImagesPullingTime="19.743514992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:33:18.19700484 +0000 UTC m=+22.738083635" watchObservedRunningTime="2026-07-09 22:33:18.197173592 +0000 UTC m=+22.738252381" Jul 09 22:33:18.319046 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:18.319017 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:18.418441 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:18.418405 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:18.703143 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:18.703047 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:18.759981 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:18.759938 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:18.950998 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:18.950881 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T22:33:18.125608287Z","UUID":"901e9f9c-c4da-4bcd-bd4a-68606237e00f","Handler":null,"Name":"","Endpoint":""} Jul 09 22:33:18.952502 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:18.952465 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 22:33:18.952638 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:18.952524 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 22:33:19.189602 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:19.189552 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" event={"ID":"967a3140-f9ca-4ae6-a61d-ba6b9824796a","Type":"ContainerStarted","Data":"e8950524e87f7c7cf14b8eefa726402d7cccf858612ef5023b09cbb56d276257"} Jul 09 22:33:19.189876 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:19.189702 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" event={"ID":"967a3140-f9ca-4ae6-a61d-ba6b9824796a","Type":"ContainerStarted","Data":"3d344e38f34f977a1202b279dea91102695296350d1237e57904253e1b09bf41"} Jul 09 22:33:19.189876 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:19.189752 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" event={"ID":"967a3140-f9ca-4ae6-a61d-ba6b9824796a","Type":"ContainerStarted","Data":"34b0cc469d1a7f5dc923b283aeb5acd18757e2681a1c477e88d2c6ee9b566749"} Jul 09 22:33:19.189876 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:19.189789 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" event={"ID":"967a3140-f9ca-4ae6-a61d-ba6b9824796a","Type":"ContainerStarted","Data":"d44a79e582877bfb69ecb434e28cd3d1e140be37610434efaa579cbcb3c56de8"} Jul 09 22:33:20.041938 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:20.041902 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:20.042463 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:20.041906 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:20.042463 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:20.042045 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:33:20.042463 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:20.042058 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:20.042463 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:20.042166 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nmjj" podUID="44657019-d54e-4ff7-8677-c4ec3d34e1f8" Jul 09 22:33:20.042463 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:20.042272 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvsts" podUID="9602778d-6209-402b-8ccf-4267f9a37ad5" Jul 09 22:33:20.195050 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:20.195015 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" event={"ID":"f61fa573-7943-469d-8262-523ad3aabb94","Type":"ContainerStarted","Data":"3245c773d6d8deb35fdcd676f00dfb806f1dc33aef0a4a0de92aa5b0d7383c19"} Jul 09 22:33:20.214231 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:20.214172 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wbr2" podStartSLOduration=3.590279642 podStartE2EDuration="24.214151147s" totalImagesPullingTime="20.623871505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:32:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:33:20.213667731 +0000 UTC m=+24.754746537" watchObservedRunningTime="2026-07-09 22:33:20.214151147 +0000 UTC m=+24.755229941" Jul 09 22:33:21.201391 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:21.201345 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" event={"ID":"967a3140-f9ca-4ae6-a61d-ba6b9824796a","Type":"ContainerStarted","Data":"39b73f2df6c2204cd51e04520af85ccc58ab3cff222cfccf7cdfe8b405a0c7ac"} Jul 09 22:33:21.391278 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:21.391250 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:21.452852 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:21.452763 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:22.042572 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:22.042540 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:22.042768 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:22.042540 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:22.042768 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:22.042661 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nmjj" podUID="44657019-d54e-4ff7-8677-c4ec3d34e1f8" Jul 09 22:33:22.042768 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:22.042673 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:22.042768 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:22.042745 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvsts" podUID="9602778d-6209-402b-8ccf-4267f9a37ad5" Jul 09 22:33:22.043002 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:22.042819 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:33:23.210138 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:23.209850 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" event={"ID":"967a3140-f9ca-4ae6-a61d-ba6b9824796a","Type":"ContainerStarted","Data":"983318b364052aa24d7fc2fb96ac10e2a38540d31320220d144034d431048373"} Jul 09 22:33:23.210882 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:23.210167 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:33:23.210882 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:23.210225 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:33:23.210882 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:23.210273 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:33:23.212027 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:23.211999 2568 generic.go:356] "Generic (PLEG): container finished" podID="ca43f479-c01f-49e4-b1fe-a36149b29f51" containerID="59b226e076c6a8a1fc4b35d48e09d5a6bd399861b8533a09f75deb9ac21cc07a" exitCode=0 Jul 09 22:33:23.212121 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:23.212042 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ddvw" event={"ID":"ca43f479-c01f-49e4-b1fe-a36149b29f51","Type":"ContainerDied","Data":"59b226e076c6a8a1fc4b35d48e09d5a6bd399861b8533a09f75deb9ac21cc07a"} Jul 09 22:33:23.226591 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:23.226566 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:33:23.226728 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:23.226673 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:33:23.237105 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:23.237036 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" podStartSLOduration=7.499502722 podStartE2EDuration="27.237021061s" totalImagesPullingTime="19.737518339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:33:23.235348751 +0000 UTC m=+27.776427564" watchObservedRunningTime="2026-07-09 22:33:23.237021061 +0000 UTC m=+27.778099855" Jul 09 22:33:24.041681 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:24.041595 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:24.041885 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:24.041596 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:24.041885 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:24.041720 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:33:24.041885 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:24.041598 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:24.041885 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:24.041811 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nmjj" podUID="44657019-d54e-4ff7-8677-c4ec3d34e1f8" Jul 09 22:33:24.041885 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:24.041857 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvsts" podUID="9602778d-6209-402b-8ccf-4267f9a37ad5" Jul 09 22:33:24.662337 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:24.662244 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9nmjj"] Jul 09 22:33:24.662816 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:24.662364 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:24.662816 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:24.662473 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nmjj" podUID="44657019-d54e-4ff7-8677-c4ec3d34e1f8" Jul 09 22:33:24.662989 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:24.662971 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fvsts"] Jul 09 22:33:24.663053 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:24.663042 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:24.663127 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:24.663112 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvsts" podUID="9602778d-6209-402b-8ccf-4267f9a37ad5" Jul 09 22:33:24.668931 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:24.668899 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5wch9"] Jul 09 22:33:24.669110 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:24.669094 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:24.669238 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:24.669216 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:33:25.218842 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:25.218810 2568 generic.go:356] "Generic (PLEG): container finished" podID="ca43f479-c01f-49e4-b1fe-a36149b29f51" containerID="08051e24853184fb33786f549584904c2f52a4699aabdb076309f92cfcdfa2e7" exitCode=0 Jul 09 22:33:25.219044 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:25.218890 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ddvw" event={"ID":"ca43f479-c01f-49e4-b1fe-a36149b29f51","Type":"ContainerDied","Data":"08051e24853184fb33786f549584904c2f52a4699aabdb076309f92cfcdfa2e7"} Jul 09 22:33:26.042837 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:26.042807 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:26.043207 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:26.042924 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:33:26.043207 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:26.042977 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:26.043207 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:26.043070 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvsts" podUID="9602778d-6209-402b-8ccf-4267f9a37ad5" Jul 09 22:33:26.422246 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:26.422081 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:26.524925 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:26.524893 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:27.041675 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:27.041598 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:27.041817 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:27.041690 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nmjj" podUID="44657019-d54e-4ff7-8677-c4ec3d34e1f8" Jul 09 22:33:27.229712 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:27.229680 2568 generic.go:356] "Generic (PLEG): container finished" podID="ca43f479-c01f-49e4-b1fe-a36149b29f51" containerID="95b1191aae162bda334aedf689c2d28bd01410c8972413f4a8199ef0037cb8ff" exitCode=0 Jul 09 22:33:27.230073 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:27.229736 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ddvw" event={"ID":"ca43f479-c01f-49e4-b1fe-a36149b29f51","Type":"ContainerDied","Data":"95b1191aae162bda334aedf689c2d28bd01410c8972413f4a8199ef0037cb8ff"} Jul 09 22:33:28.041955 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:28.041913 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:28.042158 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:28.041913 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:28.042158 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:28.042059 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:33:28.042158 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:28.042083 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvsts" podUID="9602778d-6209-402b-8ccf-4267f9a37ad5" Jul 09 22:33:29.041582 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:29.041542 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:29.042089 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:29.041656 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nmjj" podUID="44657019-d54e-4ff7-8677-c4ec3d34e1f8" Jul 09 22:33:29.734704 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:29.734612 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-9nmjj\" (UID: \"44657019-d54e-4ff7-8677-c4ec3d34e1f8\") " pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:29.734704 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:29.734666 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs\") pod \"network-metrics-daemon-5wch9\" (UID: \"941cb948-d902-4c71-abf2-52ec25863dd8\") " pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:29.734704 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:29.734689 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brwl5\" (UniqueName: \"kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5\") pod \"network-check-target-fvsts\" (UID: \"9602778d-6209-402b-8ccf-4267f9a37ad5\") " pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:29.734921 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:29.734778 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:33:29.734921 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:29.734792 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 22:33:29.734921 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:29.734807 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 22:33:29.734921 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:29.734818 2568 projected.go:196] Error preparing data for projected volume kube-api-access-brwl5 for pod openshift-network-diagnostics/network-check-target-fvsts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:33:29.734921 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:29.734777 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 22:33:29.734921 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:29.734850 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs podName:941cb948-d902-4c71-abf2-52ec25863dd8 nodeName:}" failed. No retries permitted until 2026-07-09 22:34:01.734831647 +0000 UTC m=+66.275910434 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs") pod "network-metrics-daemon-5wch9" (UID: "941cb948-d902-4c71-abf2-52ec25863dd8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:33:29.734921 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:29.734867 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5 podName:9602778d-6209-402b-8ccf-4267f9a37ad5 nodeName:}" failed. No retries permitted until 2026-07-09 22:34:01.734858604 +0000 UTC m=+66.275937383 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-brwl5" (UniqueName: "kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5") pod "network-check-target-fvsts" (UID: "9602778d-6209-402b-8ccf-4267f9a37ad5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:33:29.734921 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:29.734882 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret podName:44657019-d54e-4ff7-8677-c4ec3d34e1f8 nodeName:}" failed. No retries permitted until 2026-07-09 22:34:01.734874373 +0000 UTC m=+66.275953155 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret") pod "global-pull-secret-syncer-9nmjj" (UID: "44657019-d54e-4ff7-8677-c4ec3d34e1f8") : object "kube-system"/"original-pull-secret" not registered Jul 09 22:33:30.042643 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:30.042561 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:30.043063 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:30.042954 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:33:30.043063 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:30.043004 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:30.043176 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:30.043100 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvsts" podUID="9602778d-6209-402b-8ccf-4267f9a37ad5" Jul 09 22:33:31.042158 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.042123 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:31.042361 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:31.042250 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nmjj" podUID="44657019-d54e-4ff7-8677-c4ec3d34e1f8" Jul 09 22:33:31.718908 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.718828 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-80.ec2.internal" event="NodeReady" Jul 09 22:33:31.719292 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.718993 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 22:33:31.764201 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.764153 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mzf9t"] Jul 09 22:33:31.769387 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.769362 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dzhcj"] Jul 09 22:33:31.769560 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.769511 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mzf9t" Jul 09 22:33:31.772170 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.772142 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 22:33:31.772320 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.772158 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 22:33:31.772320 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.772201 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xdqcx\"" Jul 09 22:33:31.773007 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.772987 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dzhcj" Jul 09 22:33:31.775475 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.775453 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 22:33:31.775620 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.775483 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 22:33:31.775810 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.775745 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 22:33:31.776842 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.776821 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-twh9c\"" Jul 09 22:33:31.777905 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.777840 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mzf9t"] Jul 09 22:33:31.780912 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.780892 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dzhcj"] Jul 09 22:33:31.879379 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.879339 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:31.948384 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.948342 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e818597c-2b00-4cd2-9d5a-9af19eca0605-config-volume\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:33:31.948604 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.948433 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvmvw\" (UniqueName: \"kubernetes.io/projected/4ed629cc-bb6f-4797-893b-27ed35d60fb5-kube-api-access-dvmvw\") pod \"ingress-canary-dzhcj\" (UID: \"4ed629cc-bb6f-4797-893b-27ed35d60fb5\") " pod="openshift-ingress-canary/ingress-canary-dzhcj" Jul 09 22:33:31.948604 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.948470 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e818597c-2b00-4cd2-9d5a-9af19eca0605-tmp-dir\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:33:31.948604 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.948560 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg28s\" (UniqueName: \"kubernetes.io/projected/e818597c-2b00-4cd2-9d5a-9af19eca0605-kube-api-access-bg28s\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:33:31.948604 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.948596 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:33:31.948801 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.948650 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ed629cc-bb6f-4797-893b-27ed35d60fb5-cert\") pod \"ingress-canary-dzhcj\" (UID: \"4ed629cc-bb6f-4797-893b-27ed35d60fb5\") " pod="openshift-ingress-canary/ingress-canary-dzhcj" Jul 09 22:33:31.979544 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:31.979513 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:32.041759 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.041732 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:33:32.041862 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.041759 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:33:32.045072 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.044901 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 22:33:32.045072 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.044927 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d8l99\"" Jul 09 22:33:32.045072 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.044946 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 22:33:32.045072 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.044998 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 22:33:32.045072 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.045013 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jjswb\"" Jul 09 22:33:32.049057 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.049024 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg28s\" (UniqueName: \"kubernetes.io/projected/e818597c-2b00-4cd2-9d5a-9af19eca0605-kube-api-access-bg28s\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:33:32.049213 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.049074 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:33:32.049213 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.049134 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ed629cc-bb6f-4797-893b-27ed35d60fb5-cert\") pod \"ingress-canary-dzhcj\" (UID: \"4ed629cc-bb6f-4797-893b-27ed35d60fb5\") " pod="openshift-ingress-canary/ingress-canary-dzhcj" Jul 09 22:33:32.049213 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.049181 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e818597c-2b00-4cd2-9d5a-9af19eca0605-config-volume\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:33:32.049564 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.049241 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvmvw\" (UniqueName: \"kubernetes.io/projected/4ed629cc-bb6f-4797-893b-27ed35d60fb5-kube-api-access-dvmvw\") pod \"ingress-canary-dzhcj\" (UID: \"4ed629cc-bb6f-4797-893b-27ed35d60fb5\") " pod="openshift-ingress-canary/ingress-canary-dzhcj" Jul 09 22:33:32.049564 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.049320 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e818597c-2b00-4cd2-9d5a-9af19eca0605-tmp-dir\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:33:32.049823 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.049795 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e818597c-2b00-4cd2-9d5a-9af19eca0605-tmp-dir\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:33:32.049940 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:32.049858 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 22:33:32.050003 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:32.049940 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls podName:e818597c-2b00-4cd2-9d5a-9af19eca0605 nodeName:}" failed. No retries permitted until 2026-07-09 22:33:32.549919174 +0000 UTC m=+37.090997948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls") pod "dns-default-mzf9t" (UID: "e818597c-2b00-4cd2-9d5a-9af19eca0605") : secret "dns-default-metrics-tls" not found Jul 09 22:33:32.050215 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.050192 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e818597c-2b00-4cd2-9d5a-9af19eca0605-config-volume\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:33:32.053996 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.053963 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ed629cc-bb6f-4797-893b-27ed35d60fb5-cert\") pod \"ingress-canary-dzhcj\" (UID: \"4ed629cc-bb6f-4797-893b-27ed35d60fb5\") " pod="openshift-ingress-canary/ingress-canary-dzhcj" Jul 09 22:33:32.060626 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.060598 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg28s\" (UniqueName: \"kubernetes.io/projected/e818597c-2b00-4cd2-9d5a-9af19eca0605-kube-api-access-bg28s\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:33:32.061193 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.061167 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvmvw\" (UniqueName: \"kubernetes.io/projected/4ed629cc-bb6f-4797-893b-27ed35d60fb5-kube-api-access-dvmvw\") pod \"ingress-canary-dzhcj\" (UID: \"4ed629cc-bb6f-4797-893b-27ed35d60fb5\") " pod="openshift-ingress-canary/ingress-canary-dzhcj" Jul 09 22:33:32.088405 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.088375 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dzhcj" Jul 09 22:33:32.236731 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.236402 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dzhcj"] Jul 09 22:33:32.241169 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:33:32.241144 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed629cc_bb6f_4797_893b_27ed35d60fb5.slice/crio-3950eebba22e79888fe8446ca1182151407ca90712162e1bc41becdecb8452d2 WatchSource:0}: Error finding container 3950eebba22e79888fe8446ca1182151407ca90712162e1bc41becdecb8452d2: Status 404 returned error can't find the container with id 3950eebba22e79888fe8446ca1182151407ca90712162e1bc41becdecb8452d2 Jul 09 22:33:32.244469 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.244441 2568 generic.go:356] "Generic (PLEG): container finished" podID="ca43f479-c01f-49e4-b1fe-a36149b29f51" containerID="288bfc3b7c5023e48180a39d48dfe3fadbf0b68a0dd12fded99bb568872ff33f" exitCode=0 Jul 09 22:33:32.244581 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.244525 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ddvw" event={"ID":"ca43f479-c01f-49e4-b1fe-a36149b29f51","Type":"ContainerDied","Data":"288bfc3b7c5023e48180a39d48dfe3fadbf0b68a0dd12fded99bb568872ff33f"} Jul 09 22:33:32.553407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:32.553312 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:33:32.553575 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:32.553458 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 22:33:32.553575 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:32.553548 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls podName:e818597c-2b00-4cd2-9d5a-9af19eca0605 nodeName:}" failed. No retries permitted until 2026-07-09 22:33:33.553527392 +0000 UTC m=+38.094606186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls") pod "dns-default-mzf9t" (UID: "e818597c-2b00-4cd2-9d5a-9af19eca0605") : secret "dns-default-metrics-tls" not found Jul 09 22:33:33.041773 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:33.041741 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:33:33.045381 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:33.045357 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 22:33:33.248445 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:33.248413 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dzhcj" event={"ID":"4ed629cc-bb6f-4797-893b-27ed35d60fb5","Type":"ContainerStarted","Data":"3950eebba22e79888fe8446ca1182151407ca90712162e1bc41becdecb8452d2"} Jul 09 22:33:33.251894 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:33.251866 2568 generic.go:356] "Generic (PLEG): container finished" podID="ca43f479-c01f-49e4-b1fe-a36149b29f51" containerID="3a447756127d97e0df9666fb2a83695517091efd9009be178a3c242a55f1bf48" exitCode=0 Jul 09 22:33:33.252035 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:33.251909 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ddvw" event={"ID":"ca43f479-c01f-49e4-b1fe-a36149b29f51","Type":"ContainerDied","Data":"3a447756127d97e0df9666fb2a83695517091efd9009be178a3c242a55f1bf48"} Jul 09 22:33:33.562176 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:33.562083 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:33:33.562338 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:33.562234 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 22:33:33.562338 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:33.562306 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls podName:e818597c-2b00-4cd2-9d5a-9af19eca0605 nodeName:}" failed. No retries permitted until 2026-07-09 22:33:35.562287398 +0000 UTC m=+40.103366187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls") pod "dns-default-mzf9t" (UID: "e818597c-2b00-4cd2-9d5a-9af19eca0605") : secret "dns-default-metrics-tls" not found Jul 09 22:33:34.258990 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:34.258945 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8ddvw" event={"ID":"ca43f479-c01f-49e4-b1fe-a36149b29f51","Type":"ContainerStarted","Data":"acf2f6bbd630ea04d8b453572f541c95adcd1fadd3236bfd45475f35d98007b0"} Jul 09 22:33:34.287230 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:34.287175 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8ddvw" podStartSLOduration=7.550566486 podStartE2EDuration="38.28715001s" totalImagesPullingTime="30.736583524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:32:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:33:34.285195453 +0000 UTC m=+38.826274248" watchObservedRunningTime="2026-07-09 22:33:34.28715001 +0000 UTC m=+38.828228807" Jul 09 22:33:34.410230 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:34.410190 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:34.512829 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:34.512739 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:34.575825 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:34.575784 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:35.262986 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:35.262745 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dzhcj" event={"ID":"4ed629cc-bb6f-4797-893b-27ed35d60fb5","Type":"ContainerStarted","Data":"aabf7d2b889014700a745e7178eb04d842b8149a02f10c017af0b73d432474ce"} Jul 09 22:33:35.280061 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:35.280012 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dzhcj" podStartSLOduration=1.5315595549999999 podStartE2EDuration="4.279993727s" totalImagesPullingTime="2.748434172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:33:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:33:35.279530274 +0000 UTC m=+39.820609093" watchObservedRunningTime="2026-07-09 22:33:35.279993727 +0000 UTC m=+39.821072522" Jul 09 22:33:35.575167 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:35.575132 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:33:35.575370 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:35.575249 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 22:33:35.575370 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:35.575311 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls podName:e818597c-2b00-4cd2-9d5a-9af19eca0605 nodeName:}" failed. No retries permitted until 2026-07-09 22:33:39.575298156 +0000 UTC m=+44.116376928 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls") pod "dns-default-mzf9t" (UID: "e818597c-2b00-4cd2-9d5a-9af19eca0605") : secret "dns-default-metrics-tls" not found Jul 09 22:33:39.371428 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:39.371383 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:39.427286 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:39.427239 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:39.603084 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:39.603048 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:33:39.603243 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:39.603194 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 22:33:39.603298 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:39.603286 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls podName:e818597c-2b00-4cd2-9d5a-9af19eca0605 nodeName:}" failed. No retries permitted until 2026-07-09 22:33:47.603270912 +0000 UTC m=+52.144349685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls") pod "dns-default-mzf9t" (UID: "e818597c-2b00-4cd2-9d5a-9af19eca0605") : secret "dns-default-metrics-tls" not found Jul 09 22:33:43.745811 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:43.745738 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:43.805906 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:43.805869 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:47.665308 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:47.665265 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:33:47.665730 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:47.665391 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 22:33:47.665730 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:33:47.665442 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls podName:e818597c-2b00-4cd2-9d5a-9af19eca0605 nodeName:}" failed. No retries permitted until 2026-07-09 22:34:03.665428479 +0000 UTC m=+68.206507251 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls") pod "dns-default-mzf9t" (UID: "e818597c-2b00-4cd2-9d5a-9af19eca0605") : secret "dns-default-metrics-tls" not found Jul 09 22:33:53.389850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.389814 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr"] Jul 09 22:33:53.391624 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.391607 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.394556 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.394530 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 09 22:33:53.394698 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.394530 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 09 22:33:53.394698 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.394576 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 09 22:33:53.394698 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.394588 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 09 22:33:53.395700 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.395681 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 09 22:33:53.395817 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.395685 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 09 22:33:53.396014 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.396001 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 09 22:33:53.402394 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.402370 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr"] Jul 09 22:33:53.508762 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.508718 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12015fd4-71af-4354-b45f-fb871e9e5e05-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7544dfbf9b-ch5mr\" (UID: \"12015fd4-71af-4354-b45f-fb871e9e5e05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.508960 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.508780 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctl4z\" (UniqueName: \"kubernetes.io/projected/12015fd4-71af-4354-b45f-fb871e9e5e05-kube-api-access-ctl4z\") pod \"cluster-proxy-proxy-agent-7544dfbf9b-ch5mr\" (UID: \"12015fd4-71af-4354-b45f-fb871e9e5e05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.508960 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.508810 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/12015fd4-71af-4354-b45f-fb871e9e5e05-hub\") pod \"cluster-proxy-proxy-agent-7544dfbf9b-ch5mr\" (UID: \"12015fd4-71af-4354-b45f-fb871e9e5e05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.508960 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.508859 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/12015fd4-71af-4354-b45f-fb871e9e5e05-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7544dfbf9b-ch5mr\" (UID: \"12015fd4-71af-4354-b45f-fb871e9e5e05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.508960 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.508902 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/12015fd4-71af-4354-b45f-fb871e9e5e05-ca\") pod \"cluster-proxy-proxy-agent-7544dfbf9b-ch5mr\" (UID: \"12015fd4-71af-4354-b45f-fb871e9e5e05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.508960 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.508926 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/12015fd4-71af-4354-b45f-fb871e9e5e05-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7544dfbf9b-ch5mr\" (UID: \"12015fd4-71af-4354-b45f-fb871e9e5e05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.609618 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.609583 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctl4z\" (UniqueName: \"kubernetes.io/projected/12015fd4-71af-4354-b45f-fb871e9e5e05-kube-api-access-ctl4z\") pod \"cluster-proxy-proxy-agent-7544dfbf9b-ch5mr\" (UID: \"12015fd4-71af-4354-b45f-fb871e9e5e05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.609618 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.609621 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/12015fd4-71af-4354-b45f-fb871e9e5e05-hub\") pod \"cluster-proxy-proxy-agent-7544dfbf9b-ch5mr\" (UID: \"12015fd4-71af-4354-b45f-fb871e9e5e05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.609835 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.609661 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/12015fd4-71af-4354-b45f-fb871e9e5e05-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7544dfbf9b-ch5mr\" (UID: \"12015fd4-71af-4354-b45f-fb871e9e5e05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.609835 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.609695 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/12015fd4-71af-4354-b45f-fb871e9e5e05-ca\") pod \"cluster-proxy-proxy-agent-7544dfbf9b-ch5mr\" (UID: \"12015fd4-71af-4354-b45f-fb871e9e5e05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.609926 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.609836 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/12015fd4-71af-4354-b45f-fb871e9e5e05-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7544dfbf9b-ch5mr\" (UID: \"12015fd4-71af-4354-b45f-fb871e9e5e05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.609926 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.609885 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12015fd4-71af-4354-b45f-fb871e9e5e05-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7544dfbf9b-ch5mr\" (UID: \"12015fd4-71af-4354-b45f-fb871e9e5e05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.610389 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.610363 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/12015fd4-71af-4354-b45f-fb871e9e5e05-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7544dfbf9b-ch5mr\" (UID: \"12015fd4-71af-4354-b45f-fb871e9e5e05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.613589 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.613566 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/12015fd4-71af-4354-b45f-fb871e9e5e05-ca\") pod \"cluster-proxy-proxy-agent-7544dfbf9b-ch5mr\" (UID: \"12015fd4-71af-4354-b45f-fb871e9e5e05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.613722 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.613650 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12015fd4-71af-4354-b45f-fb871e9e5e05-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7544dfbf9b-ch5mr\" (UID: \"12015fd4-71af-4354-b45f-fb871e9e5e05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.613722 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.613667 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/12015fd4-71af-4354-b45f-fb871e9e5e05-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7544dfbf9b-ch5mr\" (UID: \"12015fd4-71af-4354-b45f-fb871e9e5e05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.613722 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.613700 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/12015fd4-71af-4354-b45f-fb871e9e5e05-hub\") pod \"cluster-proxy-proxy-agent-7544dfbf9b-ch5mr\" (UID: \"12015fd4-71af-4354-b45f-fb871e9e5e05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.619271 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.619244 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctl4z\" (UniqueName: \"kubernetes.io/projected/12015fd4-71af-4354-b45f-fb871e9e5e05-kube-api-access-ctl4z\") pod \"cluster-proxy-proxy-agent-7544dfbf9b-ch5mr\" (UID: \"12015fd4-71af-4354-b45f-fb871e9e5e05\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.713249 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.713154 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:33:53.841595 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:53.841565 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr"] Jul 09 22:33:53.844667 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:33:53.844641 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12015fd4_71af_4354_b45f_fb871e9e5e05.slice/crio-5cf6aaad032c3bc7ca372e7beacec4474a1ded25fe80b2ef22a29c6bcd52e932 WatchSource:0}: Error finding container 5cf6aaad032c3bc7ca372e7beacec4474a1ded25fe80b2ef22a29c6bcd52e932: Status 404 returned error can't find the container with id 5cf6aaad032c3bc7ca372e7beacec4474a1ded25fe80b2ef22a29c6bcd52e932 Jul 09 22:33:54.309612 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:54.309572 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" event={"ID":"12015fd4-71af-4354-b45f-fb871e9e5e05","Type":"ContainerStarted","Data":"5cf6aaad032c3bc7ca372e7beacec4474a1ded25fe80b2ef22a29c6bcd52e932"} Jul 09 22:33:55.230777 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:55.230734 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pm8p2" Jul 09 22:33:55.381162 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:55.381109 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:55.466030 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:55.465992 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:55.518194 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:55.518112 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:33:58.323848 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:58.323807 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" event={"ID":"12015fd4-71af-4354-b45f-fb871e9e5e05","Type":"ContainerStarted","Data":"4c83ffb37c58a15bf78abb4b70593bd3942f83206cf420a75dba4d2b5150d1bb"} Jul 09 22:33:58.323848 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:58.323852 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" event={"ID":"12015fd4-71af-4354-b45f-fb871e9e5e05","Type":"ContainerStarted","Data":"367f502b3133b462ab7c9a3edfeb02792ae4732731b024940476c491152a29b9"} Jul 09 22:33:58.324275 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:58.323863 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" event={"ID":"12015fd4-71af-4354-b45f-fb871e9e5e05","Type":"ContainerStarted","Data":"edf22c36c19040cf401b4038e90aa3e60e4fefd0c1107b7eb2f97d45206fc9fc"} Jul 09 22:33:58.342146 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:33:58.342096 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" podStartSLOduration=1.147560055 podStartE2EDuration="5.342082403s" totalImagesPullingTime="4.194522348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:33:58.341415833 +0000 UTC m=+62.882494627" watchObservedRunningTime="2026-07-09 22:33:58.342082403 +0000 UTC m=+62.883161196" Jul 09 22:34:01.774512 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:01.774427 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-9nmjj\" (UID: \"44657019-d54e-4ff7-8677-c4ec3d34e1f8\") " pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:34:01.775080 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:01.774547 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs\") pod \"network-metrics-daemon-5wch9\" (UID: \"941cb948-d902-4c71-abf2-52ec25863dd8\") " pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:34:01.775080 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:01.774569 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brwl5\" (UniqueName: \"kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5\") pod \"network-check-target-fvsts\" (UID: \"9602778d-6209-402b-8ccf-4267f9a37ad5\") " pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:34:01.777699 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:01.777673 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 22:34:01.777699 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:01.777687 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 22:34:01.777854 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:01.777677 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 22:34:01.785430 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:34:01.785405 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 09 22:34:01.785506 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:34:01.785479 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs podName:941cb948-d902-4c71-abf2-52ec25863dd8 nodeName:}" failed. No retries permitted until 2026-07-09 22:35:05.785454551 +0000 UTC m=+130.326533324 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs") pod "network-metrics-daemon-5wch9" (UID: "941cb948-d902-4c71-abf2-52ec25863dd8") : secret "metrics-daemon-secret" not found Jul 09 22:34:01.787683 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:01.787664 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/44657019-d54e-4ff7-8677-c4ec3d34e1f8-original-pull-secret\") pod \"global-pull-secret-syncer-9nmjj\" (UID: \"44657019-d54e-4ff7-8677-c4ec3d34e1f8\") " pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:34:01.788359 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:01.788338 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 22:34:01.799022 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:01.798989 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brwl5\" (UniqueName: \"kubernetes.io/projected/9602778d-6209-402b-8ccf-4267f9a37ad5-kube-api-access-brwl5\") pod \"network-check-target-fvsts\" (UID: \"9602778d-6209-402b-8ccf-4267f9a37ad5\") " pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:34:01.853122 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:01.853075 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nmjj" Jul 09 22:34:01.978362 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:01.978326 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9nmjj"] Jul 09 22:34:01.981443 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:34:01.981415 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44657019_d54e_4ff7_8677_c4ec3d34e1f8.slice/crio-65803206dac221c138737caf0a6fe303dd2ca9069117317538bae07ecb95cb95 WatchSource:0}: Error finding container 65803206dac221c138737caf0a6fe303dd2ca9069117317538bae07ecb95cb95: Status 404 returned error can't find the container with id 65803206dac221c138737caf0a6fe303dd2ca9069117317538bae07ecb95cb95 Jul 09 22:34:02.072603 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:02.072580 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d8l99\"" Jul 09 22:34:02.079971 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:02.079941 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:34:02.207815 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:02.205747 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fvsts"] Jul 09 22:34:02.211345 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:34:02.211320 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9602778d_6209_402b_8ccf_4267f9a37ad5.slice/crio-097938de19a7a88caa2aa1c55abfc4d1631055000fe7f9ae6c00a171ec180faa WatchSource:0}: Error finding container 097938de19a7a88caa2aa1c55abfc4d1631055000fe7f9ae6c00a171ec180faa: Status 404 returned error can't find the container with id 097938de19a7a88caa2aa1c55abfc4d1631055000fe7f9ae6c00a171ec180faa Jul 09 22:34:02.333565 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:02.333439 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fvsts" event={"ID":"9602778d-6209-402b-8ccf-4267f9a37ad5","Type":"ContainerStarted","Data":"097938de19a7a88caa2aa1c55abfc4d1631055000fe7f9ae6c00a171ec180faa"} Jul 09 22:34:02.334563 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:02.334540 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9nmjj" event={"ID":"44657019-d54e-4ff7-8677-c4ec3d34e1f8","Type":"ContainerStarted","Data":"65803206dac221c138737caf0a6fe303dd2ca9069117317538bae07ecb95cb95"} Jul 09 22:34:03.687678 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:03.687638 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:34:03.688155 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:34:03.687784 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 22:34:03.688155 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:34:03.687860 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls podName:e818597c-2b00-4cd2-9d5a-9af19eca0605 nodeName:}" failed. No retries permitted until 2026-07-09 22:34:35.687839446 +0000 UTC m=+100.228918220 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls") pod "dns-default-mzf9t" (UID: "e818597c-2b00-4cd2-9d5a-9af19eca0605") : secret "dns-default-metrics-tls" not found Jul 09 22:34:06.846897 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:06.846856 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:34:07.355922 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:07.355824 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9nmjj" event={"ID":"44657019-d54e-4ff7-8677-c4ec3d34e1f8","Type":"ContainerStarted","Data":"4d2c26887a1e44a193b05aee823056140d27fd59d5e9309c8ad82480c377b005"} Jul 09 22:34:07.357152 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:07.357128 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fvsts" event={"ID":"9602778d-6209-402b-8ccf-4267f9a37ad5","Type":"ContainerStarted","Data":"8cc7eccf75a5acf111546e98f31740b59b9ef95fece3776993fd1a2c8ce3af79"} Jul 09 22:34:07.357268 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:07.357255 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:34:07.374032 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:07.373985 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9nmjj" podStartSLOduration=66.33430744099999 podStartE2EDuration="1m11.373967836s" totalImagesPullingTime="5.039660395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:34:07.372238323 +0000 UTC m=+71.913317118" watchObservedRunningTime="2026-07-09 22:34:07.373967836 +0000 UTC m=+71.915046630" Jul 09 22:34:28.277831 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:28.277708 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:34:28.339710 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:28.339675 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:34:35.710031 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:35.709976 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:34:35.710478 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:34:35.710134 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 22:34:35.710478 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:34:35.710201 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls podName:e818597c-2b00-4cd2-9d5a-9af19eca0605 nodeName:}" failed. No retries permitted until 2026-07-09 22:35:39.710184047 +0000 UTC m=+164.251262824 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls") pod "dns-default-mzf9t" (UID: "e818597c-2b00-4cd2-9d5a-9af19eca0605") : secret "dns-default-metrics-tls" not found Jul 09 22:34:38.362356 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:38.362327 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fvsts" Jul 09 22:34:38.379808 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:34:38.379747 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fvsts" podStartSLOduration=97.570802578 podStartE2EDuration="1m42.379730268s" totalImagesPullingTime="4.80892769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:34:07.391059878 +0000 UTC m=+71.932138718" watchObservedRunningTime="2026-07-09 22:34:38.379730268 +0000 UTC m=+102.920809055" Jul 09 22:35:05.812933 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:35:05.812894 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs\") pod \"network-metrics-daemon-5wch9\" (UID: \"941cb948-d902-4c71-abf2-52ec25863dd8\") " pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:35:05.813428 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:35:05.813044 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 09 22:35:05.813428 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:35:05.813113 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs podName:941cb948-d902-4c71-abf2-52ec25863dd8 nodeName:}" failed. No retries permitted until 2026-07-09 22:37:07.813096205 +0000 UTC m=+252.354174978 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs") pod "network-metrics-daemon-5wch9" (UID: "941cb948-d902-4c71-abf2-52ec25863dd8") : secret "metrics-daemon-secret" not found Jul 09 22:35:34.782566 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:35:34.782483 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mzf9t" podUID="e818597c-2b00-4cd2-9d5a-9af19eca0605" Jul 09 22:35:35.053045 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:35:35.052954 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:35:35.599105 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:35:35.599080 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mzf9t" Jul 09 22:35:39.741583 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:35:39.741546 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:35:39.742178 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:35:39.741725 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 22:35:39.742178 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:35:39.741825 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls podName:e818597c-2b00-4cd2-9d5a-9af19eca0605 nodeName:}" failed. No retries permitted until 2026-07-09 22:37:41.741802452 +0000 UTC m=+286.282881237 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls") pod "dns-default-mzf9t" (UID: "e818597c-2b00-4cd2-9d5a-9af19eca0605") : secret "dns-default-metrics-tls" not found Jul 09 22:35:50.042357 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:35:50.042269 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:36:22.283554 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:22.283513 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:36:53.400053 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.400019 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-j22gw"] Jul 09 22:36:53.402891 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.402865 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-db664c774-kspt4"] Jul 09 22:36:53.403009 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.402985 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-j22gw" Jul 09 22:36:53.405442 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.405422 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-55btr\"" Jul 09 22:36:53.405769 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.405748 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.408193 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.408172 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6gjz6\"" Jul 09 22:36:53.408713 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.408695 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 22:36:53.408839 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.408823 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 22:36:53.408839 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.408833 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 22:36:53.414692 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.414663 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 22:36:53.416065 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.416033 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-j22gw"] Jul 09 22:36:53.421926 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.421900 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-db664c774-kspt4"] Jul 09 22:36:53.537312 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.537268 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddlbh\" (UniqueName: \"kubernetes.io/projected/1f24a765-beba-4490-a4a8-7188caf8ff41-kube-api-access-ddlbh\") pod \"network-check-source-989f8b8dc-j22gw\" (UID: \"1f24a765-beba-4490-a4a8-7188caf8ff41\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-j22gw" Jul 09 22:36:53.537517 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.537340 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab92fa47-2296-41b9-9636-b98ffd793b8a-ca-trust-extracted\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.537517 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.537365 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab92fa47-2296-41b9-9636-b98ffd793b8a-image-registry-private-configuration\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.537517 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.537381 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-certificates\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.537517 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.537416 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.537517 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.537432 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab92fa47-2296-41b9-9636-b98ffd793b8a-trusted-ca\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.537682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.537523 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4pxt\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-kube-api-access-t4pxt\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.537682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.537579 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab92fa47-2296-41b9-9636-b98ffd793b8a-installation-pull-secrets\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.537682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.537661 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-bound-sa-token\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.552428 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.552388 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:36:53.638192 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.638158 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.638192 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.638196 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab92fa47-2296-41b9-9636-b98ffd793b8a-trusted-ca\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.638385 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.638230 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4pxt\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-kube-api-access-t4pxt\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.638385 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:53.638321 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 22:36:53.638385 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:53.638335 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-db664c774-kspt4: secret "image-registry-tls" not found Jul 09 22:36:53.638565 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:53.638396 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls podName:ab92fa47-2296-41b9-9636-b98ffd793b8a nodeName:}" failed. No retries permitted until 2026-07-09 22:36:54.138375729 +0000 UTC m=+238.679454519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls") pod "image-registry-db664c774-kspt4" (UID: "ab92fa47-2296-41b9-9636-b98ffd793b8a") : secret "image-registry-tls" not found Jul 09 22:36:53.638565 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.638415 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab92fa47-2296-41b9-9636-b98ffd793b8a-installation-pull-secrets\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.638565 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.638470 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-bound-sa-token\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.638565 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.638537 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddlbh\" (UniqueName: \"kubernetes.io/projected/1f24a765-beba-4490-a4a8-7188caf8ff41-kube-api-access-ddlbh\") pod \"network-check-source-989f8b8dc-j22gw\" (UID: \"1f24a765-beba-4490-a4a8-7188caf8ff41\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-j22gw" Jul 09 22:36:53.638565 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.638561 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab92fa47-2296-41b9-9636-b98ffd793b8a-ca-trust-extracted\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.638818 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.638588 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab92fa47-2296-41b9-9636-b98ffd793b8a-image-registry-private-configuration\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.638818 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.638612 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-certificates\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.638953 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.638932 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab92fa47-2296-41b9-9636-b98ffd793b8a-ca-trust-extracted\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.639135 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.639110 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-certificates\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.639391 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.639370 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab92fa47-2296-41b9-9636-b98ffd793b8a-trusted-ca\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.640884 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.640866 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab92fa47-2296-41b9-9636-b98ffd793b8a-image-registry-private-configuration\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.640938 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.640928 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab92fa47-2296-41b9-9636-b98ffd793b8a-installation-pull-secrets\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.647879 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.647853 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-bound-sa-token\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.648148 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.648130 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4pxt\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-kube-api-access-t4pxt\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:53.648212 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.648192 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddlbh\" (UniqueName: \"kubernetes.io/projected/1f24a765-beba-4490-a4a8-7188caf8ff41-kube-api-access-ddlbh\") pod \"network-check-source-989f8b8dc-j22gw\" (UID: \"1f24a765-beba-4490-a4a8-7188caf8ff41\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-j22gw" Jul 09 22:36:53.651970 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.651906 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:36:53.713960 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.713916 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-j22gw" Jul 09 22:36:53.841530 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:53.841474 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-j22gw"] Jul 09 22:36:53.844105 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:36:53.844075 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f24a765_beba_4490_a4a8_7188caf8ff41.slice/crio-4776424f3ba62c101d9b8665f840248235b243af47ecc407ae08567cf44e56a8 WatchSource:0}: Error finding container 4776424f3ba62c101d9b8665f840248235b243af47ecc407ae08567cf44e56a8: Status 404 returned error can't find the container with id 4776424f3ba62c101d9b8665f840248235b243af47ecc407ae08567cf44e56a8 Jul 09 22:36:54.141816 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:54.141774 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:54.141989 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:54.141909 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 22:36:54.141989 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:54.141930 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-db664c774-kspt4: secret "image-registry-tls" not found Jul 09 22:36:54.142076 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:54.141999 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls podName:ab92fa47-2296-41b9-9636-b98ffd793b8a nodeName:}" failed. No retries permitted until 2026-07-09 22:36:55.141980333 +0000 UTC m=+239.683059126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls") pod "image-registry-db664c774-kspt4" (UID: "ab92fa47-2296-41b9-9636-b98ffd793b8a") : secret "image-registry-tls" not found Jul 09 22:36:54.823373 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:54.823332 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-j22gw" event={"ID":"1f24a765-beba-4490-a4a8-7188caf8ff41","Type":"ContainerStarted","Data":"ee3a35b7f7c11b6d0e62345576574d6e516dad4b562a75a8eacf7c41e438d850"} Jul 09 22:36:54.823827 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:54.823394 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-j22gw" event={"ID":"1f24a765-beba-4490-a4a8-7188caf8ff41","Type":"ContainerStarted","Data":"4776424f3ba62c101d9b8665f840248235b243af47ecc407ae08567cf44e56a8"} Jul 09 22:36:54.839118 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:54.839063 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-j22gw" podStartSLOduration=1.839044444 podStartE2EDuration="1.839044444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:36:54.838785517 +0000 UTC m=+239.379864327" watchObservedRunningTime="2026-07-09 22:36:54.839044444 +0000 UTC m=+239.380123239" Jul 09 22:36:55.149246 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:55.149149 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:55.149401 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:55.149331 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 22:36:55.149401 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:55.149354 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-db664c774-kspt4: secret "image-registry-tls" not found Jul 09 22:36:55.149484 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:55.149423 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls podName:ab92fa47-2296-41b9-9636-b98ffd793b8a nodeName:}" failed. No retries permitted until 2026-07-09 22:36:57.149402284 +0000 UTC m=+241.690481060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls") pod "image-registry-db664c774-kspt4" (UID: "ab92fa47-2296-41b9-9636-b98ffd793b8a") : secret "image-registry-tls" not found Jul 09 22:36:55.914284 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:55.914250 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:36:55.969611 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:55.969582 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:36:57.080166 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:57.080130 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:36:57.164560 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:57.164526 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:36:57.164698 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:57.164634 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 22:36:57.164698 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:57.164646 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-db664c774-kspt4: secret "image-registry-tls" not found Jul 09 22:36:57.164698 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:57.164691 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls podName:ab92fa47-2296-41b9-9636-b98ffd793b8a nodeName:}" failed. No retries permitted until 2026-07-09 22:37:01.164677831 +0000 UTC m=+245.705756603 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls") pod "image-registry-db664c774-kspt4" (UID: "ab92fa47-2296-41b9-9636-b98ffd793b8a") : secret "image-registry-tls" not found Jul 09 22:36:57.181654 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:57.181623 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:36:57.804081 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:57.804050 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:36:57.886342 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:57.886303 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:36:58.030613 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.030585 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p2c2c"] Jul 09 22:36:58.033636 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.033620 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:36:58.036467 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.036442 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 22:36:58.037171 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.037151 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 22:36:58.037855 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.037838 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pvkvm\"" Jul 09 22:36:58.038589 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.038560 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 22:36:58.038589 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.038560 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 22:36:58.049216 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.049187 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p2c2c"] Jul 09 22:36:58.061432 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.061395 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:36:58.173559 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.173518 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3658c581-11c5-4dc1-b603-ea68054b7373-crio-socket\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:36:58.173559 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.173561 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3658c581-11c5-4dc1-b603-ea68054b7373-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:36:58.173978 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.173594 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3658c581-11c5-4dc1-b603-ea68054b7373-data-volume\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:36:58.173978 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.173613 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2j9x\" (UniqueName: \"kubernetes.io/projected/3658c581-11c5-4dc1-b603-ea68054b7373-kube-api-access-n2j9x\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:36:58.173978 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.173666 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3658c581-11c5-4dc1-b603-ea68054b7373-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:36:58.274478 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.274446 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3658c581-11c5-4dc1-b603-ea68054b7373-data-volume\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:36:58.274478 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.274479 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2j9x\" (UniqueName: \"kubernetes.io/projected/3658c581-11c5-4dc1-b603-ea68054b7373-kube-api-access-n2j9x\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:36:58.274724 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.274580 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3658c581-11c5-4dc1-b603-ea68054b7373-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:36:58.274724 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.274617 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3658c581-11c5-4dc1-b603-ea68054b7373-crio-socket\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:36:58.274724 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.274633 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3658c581-11c5-4dc1-b603-ea68054b7373-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:36:58.274724 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:58.274718 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 22:36:58.274892 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.274733 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3658c581-11c5-4dc1-b603-ea68054b7373-crio-socket\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:36:58.274892 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:58.274727 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 22:36:58.274892 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:58.274793 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3658c581-11c5-4dc1-b603-ea68054b7373-kube-rbac-proxy-cm podName:3658c581-11c5-4dc1-b603-ea68054b7373 nodeName:}" failed. No retries permitted until 2026-07-09 22:36:58.77477293 +0000 UTC m=+243.315851718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3658c581-11c5-4dc1-b603-ea68054b7373-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2c2c" (UID: "3658c581-11c5-4dc1-b603-ea68054b7373") : configmap "kube-rbac-proxy" not found Jul 09 22:36:58.274892 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:58.274813 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3658c581-11c5-4dc1-b603-ea68054b7373-insights-runtime-extractor-tls podName:3658c581-11c5-4dc1-b603-ea68054b7373 nodeName:}" failed. No retries permitted until 2026-07-09 22:36:58.77480327 +0000 UTC m=+243.315882048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3658c581-11c5-4dc1-b603-ea68054b7373-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2c2c" (UID: "3658c581-11c5-4dc1-b603-ea68054b7373") : secret "insights-runtime-extractor-tls" not found Jul 09 22:36:58.274892 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.274792 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3658c581-11c5-4dc1-b603-ea68054b7373-data-volume\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:36:58.283366 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.283344 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2j9x\" (UniqueName: \"kubernetes.io/projected/3658c581-11c5-4dc1-b603-ea68054b7373-kube-api-access-n2j9x\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:36:58.778790 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.778750 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3658c581-11c5-4dc1-b603-ea68054b7373-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:36:58.778981 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.778803 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3658c581-11c5-4dc1-b603-ea68054b7373-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:36:58.778981 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:58.778873 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 22:36:58.778981 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:58.778910 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 22:36:58.778981 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:58.778943 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3658c581-11c5-4dc1-b603-ea68054b7373-kube-rbac-proxy-cm podName:3658c581-11c5-4dc1-b603-ea68054b7373 nodeName:}" failed. No retries permitted until 2026-07-09 22:36:59.778923651 +0000 UTC m=+244.320002424 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3658c581-11c5-4dc1-b603-ea68054b7373-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2c2c" (UID: "3658c581-11c5-4dc1-b603-ea68054b7373") : configmap "kube-rbac-proxy" not found Jul 09 22:36:58.778981 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:58.778973 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3658c581-11c5-4dc1-b603-ea68054b7373-insights-runtime-extractor-tls podName:3658c581-11c5-4dc1-b603-ea68054b7373 nodeName:}" failed. No retries permitted until 2026-07-09 22:36:59.778962514 +0000 UTC m=+244.320041287 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3658c581-11c5-4dc1-b603-ea68054b7373-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2c2c" (UID: "3658c581-11c5-4dc1-b603-ea68054b7373") : secret "insights-runtime-extractor-tls" not found Jul 09 22:36:58.990349 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:58.990316 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:36:59.085931 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:59.085896 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:36:59.185575 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:59.185542 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:36:59.428645 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:59.428554 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:36:59.527141 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:59.527108 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:36:59.787014 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:59.786917 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3658c581-11c5-4dc1-b603-ea68054b7373-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:36:59.787170 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:36:59.787026 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3658c581-11c5-4dc1-b603-ea68054b7373-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:36:59.787170 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:59.787067 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 22:36:59.787170 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:59.787102 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 22:36:59.787170 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:59.787129 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3658c581-11c5-4dc1-b603-ea68054b7373-insights-runtime-extractor-tls podName:3658c581-11c5-4dc1-b603-ea68054b7373 nodeName:}" failed. No retries permitted until 2026-07-09 22:37:01.787114435 +0000 UTC m=+246.328193210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3658c581-11c5-4dc1-b603-ea68054b7373-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2c2c" (UID: "3658c581-11c5-4dc1-b603-ea68054b7373") : secret "insights-runtime-extractor-tls" not found Jul 09 22:36:59.787170 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:36:59.787147 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3658c581-11c5-4dc1-b603-ea68054b7373-kube-rbac-proxy-cm podName:3658c581-11c5-4dc1-b603-ea68054b7373 nodeName:}" failed. No retries permitted until 2026-07-09 22:37:01.787135703 +0000 UTC m=+246.328214475 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3658c581-11c5-4dc1-b603-ea68054b7373-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2c2c" (UID: "3658c581-11c5-4dc1-b603-ea68054b7373") : configmap "kube-rbac-proxy" not found Jul 09 22:37:00.206625 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:00.206597 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sgpbm_85d0d04e-7886-4803-82a9-980ae8b0d714/dns-node-resolver/0.log" Jul 09 22:37:00.438409 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:00.438375 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:01.198345 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:01.198300 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:37:01.198568 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:01.198426 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 22:37:01.198568 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:01.198438 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-db664c774-kspt4: secret "image-registry-tls" not found Jul 09 22:37:01.198568 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:01.198485 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls podName:ab92fa47-2296-41b9-9636-b98ffd793b8a nodeName:}" failed. No retries permitted until 2026-07-09 22:37:09.198470891 +0000 UTC m=+253.739549663 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls") pod "image-registry-db664c774-kspt4" (UID: "ab92fa47-2296-41b9-9636-b98ffd793b8a") : secret "image-registry-tls" not found Jul 09 22:37:01.206847 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:01.206810 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vsrl6_b8d3fdbc-afb4-486f-a376-b4e70e32bdd8/node-ca/0.log" Jul 09 22:37:01.801977 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:01.801942 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3658c581-11c5-4dc1-b603-ea68054b7373-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:37:01.802158 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:01.801985 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3658c581-11c5-4dc1-b603-ea68054b7373-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:37:01.802158 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:01.802071 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 22:37:01.802158 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:01.802092 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 22:37:01.802158 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:01.802138 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3658c581-11c5-4dc1-b603-ea68054b7373-kube-rbac-proxy-cm podName:3658c581-11c5-4dc1-b603-ea68054b7373 nodeName:}" failed. No retries permitted until 2026-07-09 22:37:05.802118813 +0000 UTC m=+250.343197603 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3658c581-11c5-4dc1-b603-ea68054b7373-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2c2c" (UID: "3658c581-11c5-4dc1-b603-ea68054b7373") : configmap "kube-rbac-proxy" not found Jul 09 22:37:01.802158 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:01.802154 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3658c581-11c5-4dc1-b603-ea68054b7373-insights-runtime-extractor-tls podName:3658c581-11c5-4dc1-b603-ea68054b7373 nodeName:}" failed. No retries permitted until 2026-07-09 22:37:05.802147177 +0000 UTC m=+250.343225948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3658c581-11c5-4dc1-b603-ea68054b7373-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2c2c" (UID: "3658c581-11c5-4dc1-b603-ea68054b7373") : secret "insights-runtime-extractor-tls" not found Jul 09 22:37:01.805785 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:01.805755 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dzhcj_4ed629cc-bb6f-4797-893b-27ed35d60fb5/serve-healthcheck-canary/0.log" Jul 09 22:37:03.387950 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:03.387921 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:03.640423 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:03.640335 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:03.736949 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:03.736910 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:03.928234 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:03.928146 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:04.029322 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:04.029291 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:04.129555 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:04.129523 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:04.375961 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:04.375930 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:05.123394 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:05.123361 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:05.521312 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:05.521212 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:05.833012 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:05.832977 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3658c581-11c5-4dc1-b603-ea68054b7373-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:37:05.833169 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:05.833028 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3658c581-11c5-4dc1-b603-ea68054b7373-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:37:05.833169 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:05.833105 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 22:37:05.833169 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:05.833158 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 22:37:05.833267 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:05.833179 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3658c581-11c5-4dc1-b603-ea68054b7373-kube-rbac-proxy-cm podName:3658c581-11c5-4dc1-b603-ea68054b7373 nodeName:}" failed. No retries permitted until 2026-07-09 22:37:13.833160535 +0000 UTC m=+258.374239308 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3658c581-11c5-4dc1-b603-ea68054b7373-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2c2c" (UID: "3658c581-11c5-4dc1-b603-ea68054b7373") : configmap "kube-rbac-proxy" not found Jul 09 22:37:05.833267 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:05.833202 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3658c581-11c5-4dc1-b603-ea68054b7373-insights-runtime-extractor-tls podName:3658c581-11c5-4dc1-b603-ea68054b7373 nodeName:}" failed. No retries permitted until 2026-07-09 22:37:13.833190126 +0000 UTC m=+258.374268898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3658c581-11c5-4dc1-b603-ea68054b7373-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2c2c" (UID: "3658c581-11c5-4dc1-b603-ea68054b7373") : secret "insights-runtime-extractor-tls" not found Jul 09 22:37:05.924748 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:05.924715 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:06.316188 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:06.316154 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:06.718000 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:06.717900 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:06.927900 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:06.927869 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:07.518010 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:07.517981 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:07.575452 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:07.575421 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:07.849968 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:07.849940 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs\") pod \"network-metrics-daemon-5wch9\" (UID: \"941cb948-d902-4c71-abf2-52ec25863dd8\") " pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:37:07.850129 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:07.850061 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 09 22:37:07.850129 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:07.850116 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs podName:941cb948-d902-4c71-abf2-52ec25863dd8 nodeName:}" failed. No retries permitted until 2026-07-09 22:39:09.850102251 +0000 UTC m=+374.391181023 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs") pod "network-metrics-daemon-5wch9" (UID: "941cb948-d902-4c71-abf2-52ec25863dd8") : secret "metrics-daemon-secret" not found Jul 09 22:37:07.925723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:07.925687 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:08.317961 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:08.317927 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:08.518811 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:08.518769 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:08.919698 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:08.919662 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:09.262241 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:09.262153 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:37:09.264498 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:09.264467 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls\") pod \"image-registry-db664c774-kspt4\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:37:09.324258 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:09.324229 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6gjz6\"" Jul 09 22:37:09.331913 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:09.331885 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:37:09.468118 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:09.468087 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-db664c774-kspt4"] Jul 09 22:37:09.471414 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:37:09.471387 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab92fa47_2296_41b9_9636_b98ffd793b8a.slice/crio-a170c3ecdfb65b83935e67251f4b66eaee86b29c02360162c6dfac5ebd6fc449 WatchSource:0}: Error finding container a170c3ecdfb65b83935e67251f4b66eaee86b29c02360162c6dfac5ebd6fc449: Status 404 returned error can't find the container with id a170c3ecdfb65b83935e67251f4b66eaee86b29c02360162c6dfac5ebd6fc449 Jul 09 22:37:09.525381 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:09.525347 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:09.864950 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:09.864920 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-db664c774-kspt4" event={"ID":"ab92fa47-2296-41b9-9636-b98ffd793b8a","Type":"ContainerStarted","Data":"89d6f34b440478bb65e7ee7b168ab71229e3bdb477074acdcdf90173c225141c"} Jul 09 22:37:09.865122 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:09.864967 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-db664c774-kspt4" event={"ID":"ab92fa47-2296-41b9-9636-b98ffd793b8a","Type":"ContainerStarted","Data":"a170c3ecdfb65b83935e67251f4b66eaee86b29c02360162c6dfac5ebd6fc449"} Jul 09 22:37:09.865122 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:09.865063 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:37:09.886483 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:09.886433 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-db664c774-kspt4" podStartSLOduration=16.886414837 podStartE2EDuration="16.886414837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:37:09.884844263 +0000 UTC m=+254.425923057" watchObservedRunningTime="2026-07-09 22:37:09.886414837 +0000 UTC m=+254.427493630" Jul 09 22:37:10.118708 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:10.118618 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:10.177311 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:10.177272 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:10.318410 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:10.318379 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:10.378338 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:10.378235 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:10.522612 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:10.522583 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:10.915594 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:10.915554 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:11.322023 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:11.321993 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:11.924468 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:11.924423 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:11.982013 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:11.981981 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:12.318909 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:12.318878 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:12.521766 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:12.521733 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:12.730939 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:12.730849 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:13.303399 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:13.303364 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:13.404463 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:13.404374 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:13.532847 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:13.532815 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:13.726740 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:13.726704 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:13.901315 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:13.901271 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3658c581-11c5-4dc1-b603-ea68054b7373-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:37:13.901533 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:13.901334 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3658c581-11c5-4dc1-b603-ea68054b7373-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:37:13.901856 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:13.901837 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3658c581-11c5-4dc1-b603-ea68054b7373-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:37:13.903660 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:13.903640 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3658c581-11c5-4dc1-b603-ea68054b7373-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2c2c\" (UID: \"3658c581-11c5-4dc1-b603-ea68054b7373\") " pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:37:13.942719 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:13.942681 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p2c2c" Jul 09 22:37:14.074380 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:14.074346 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p2c2c"] Jul 09 22:37:14.077547 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:37:14.077514 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3658c581_11c5_4dc1_b603_ea68054b7373.slice/crio-5689276f618d028cc9d246b6c76a40c884e9b0c57f0887f55ce41a8fb20a836f WatchSource:0}: Error finding container 5689276f618d028cc9d246b6c76a40c884e9b0c57f0887f55ce41a8fb20a836f: Status 404 returned error can't find the container with id 5689276f618d028cc9d246b6c76a40c884e9b0c57f0887f55ce41a8fb20a836f Jul 09 22:37:14.120959 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:14.120926 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:14.322278 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:14.322240 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:14.881957 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:14.881878 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2c2c" event={"ID":"3658c581-11c5-4dc1-b603-ea68054b7373","Type":"ContainerStarted","Data":"5689276f618d028cc9d246b6c76a40c884e9b0c57f0887f55ce41a8fb20a836f"} Jul 09 22:37:17.031268 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:17.031232 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:17.132176 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:17.132141 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:17.197426 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:17.197387 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:17.326392 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:17.326353 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:17.707121 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:17.707030 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:18.714289 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:18.714245 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:18.815923 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:18.815892 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:18.896263 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:18.896222 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2c2c" event={"ID":"3658c581-11c5-4dc1-b603-ea68054b7373","Type":"ContainerStarted","Data":"e7639aa87bec112510b5ca3f1a0e793532a5dc437b7abd17eb9c1281bce8d246"} Jul 09 22:37:18.896457 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:18.896296 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2c2c" event={"ID":"3658c581-11c5-4dc1-b603-ea68054b7373","Type":"ContainerStarted","Data":"4222195a33f005f28b499cecf6e9028fd819f23866aced2c95137f63724b9f6b"} Jul 09 22:37:18.921943 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:18.921906 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:18.977325 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:18.977278 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:19.037222 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:19.037151 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:19.138911 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:19.138826 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:21.576008 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:21.575971 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-db664c774-kspt4"] Jul 09 22:37:21.581368 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:21.581336 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:21.688213 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:21.688182 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:21.787121 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:21.787091 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:21.907765 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:21.907682 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2c2c" event={"ID":"3658c581-11c5-4dc1-b603-ea68054b7373","Type":"ContainerStarted","Data":"745e53849252b888c091f6ca6846a86d09294e9731a309d6dbf188898038eda3"} Jul 09 22:37:21.925411 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:21.925367 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p2c2c" podStartSLOduration=17.165642764 podStartE2EDuration="23.925352301s" totalImagesPullingTime="6.759709537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:36:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:37:21.925134297 +0000 UTC m=+266.466213102" watchObservedRunningTime="2026-07-09 22:37:21.925352301 +0000 UTC m=+266.466431095" Jul 09 22:37:28.079852 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:28.079806 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:28.250593 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:28.250557 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tj8f6"] Jul 09 22:37:28.253658 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:28.253642 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tj8f6" Jul 09 22:37:28.256676 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:28.256649 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 09 22:37:28.256810 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:28.256739 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2n4lg\"" Jul 09 22:37:28.263470 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:28.263446 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tj8f6"] Jul 09 22:37:28.312461 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:28.312421 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/43700bb9-ff9c-44c6-b7c4-5f83e5bb8d56-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-tj8f6\" (UID: \"43700bb9-ff9c-44c6-b7c4-5f83e5bb8d56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tj8f6" Jul 09 22:37:28.413908 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:28.413809 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/43700bb9-ff9c-44c6-b7c4-5f83e5bb8d56-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-tj8f6\" (UID: \"43700bb9-ff9c-44c6-b7c4-5f83e5bb8d56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tj8f6" Jul 09 22:37:28.414064 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:28.413958 2568 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 09 22:37:28.414064 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:28.414023 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43700bb9-ff9c-44c6-b7c4-5f83e5bb8d56-tls-certificates podName:43700bb9-ff9c-44c6-b7c4-5f83e5bb8d56 nodeName:}" failed. No retries permitted until 2026-07-09 22:37:28.914007819 +0000 UTC m=+273.455086591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/43700bb9-ff9c-44c6-b7c4-5f83e5bb8d56-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-tj8f6" (UID: "43700bb9-ff9c-44c6-b7c4-5f83e5bb8d56") : secret "prometheus-operator-admission-webhook-tls" not found Jul 09 22:37:28.917655 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:28.917608 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/43700bb9-ff9c-44c6-b7c4-5f83e5bb8d56-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-tj8f6\" (UID: \"43700bb9-ff9c-44c6-b7c4-5f83e5bb8d56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tj8f6" Jul 09 22:37:28.920055 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:28.920030 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/43700bb9-ff9c-44c6-b7c4-5f83e5bb8d56-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-tj8f6\" (UID: \"43700bb9-ff9c-44c6-b7c4-5f83e5bb8d56\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tj8f6" Jul 09 22:37:29.162095 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:29.162058 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tj8f6" Jul 09 22:37:29.311248 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:29.311217 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tj8f6"] Jul 09 22:37:29.314407 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:37:29.314379 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43700bb9_ff9c_44c6_b7c4_5f83e5bb8d56.slice/crio-a3d970a6b1b84bf3f21d58d11bf10110761ebaa1cb0dafda64cea64ff8f2bcf2 WatchSource:0}: Error finding container a3d970a6b1b84bf3f21d58d11bf10110761ebaa1cb0dafda64cea64ff8f2bcf2: Status 404 returned error can't find the container with id a3d970a6b1b84bf3f21d58d11bf10110761ebaa1cb0dafda64cea64ff8f2bcf2 Jul 09 22:37:29.335430 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:29.335387 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:29.934062 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:29.934029 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tj8f6" event={"ID":"43700bb9-ff9c-44c6-b7c4-5f83e5bb8d56","Type":"ContainerStarted","Data":"a3d970a6b1b84bf3f21d58d11bf10110761ebaa1cb0dafda64cea64ff8f2bcf2"} Jul 09 22:37:30.937880 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:30.937847 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tj8f6" event={"ID":"43700bb9-ff9c-44c6-b7c4-5f83e5bb8d56","Type":"ContainerStarted","Data":"d1650cf4b52f232780c1ae7a06718f5fc0609b30b9c4698cd24cb726a6b64210"} Jul 09 22:37:30.938351 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:30.938049 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tj8f6" Jul 09 22:37:30.942743 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:30.942722 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tj8f6" Jul 09 22:37:30.955968 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:30.955930 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-tj8f6" podStartSLOduration=1.8753247480000002 podStartE2EDuration="2.955918004s" totalImagesPullingTime="1.080593256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:37:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:37:30.95451948 +0000 UTC m=+275.495598275" watchObservedRunningTime="2026-07-09 22:37:30.955918004 +0000 UTC m=+275.496996797" Jul 09 22:37:31.298260 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:31.298170 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:31.381992 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:31.381958 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:31.582670 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:31.582644 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:37:31.858704 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:31.858621 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:33.074071 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:33.074035 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:33.130608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:33.130573 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:35.051287 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:35.051252 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:35.103942 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:35.103887 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:35.375887 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:35.375851 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:35.476935 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:35.476896 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:35.577685 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:35.577655 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:35.678269 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:35.678195 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:36.019309 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:36.019216 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:36.121154 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:36.121117 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:36.180989 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:36.180959 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:36.239843 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:36.239809 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:37.437659 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.437627 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:37.666323 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.666286 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:37.707501 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.707405 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr"] Jul 09 22:37:37.715514 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.715474 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dw4vg"] Jul 09 22:37:37.715661 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.715483 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:37.718120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.718095 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 09 22:37:37.719081 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.719064 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.719802 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.719784 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 22:37:37.719802 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.719796 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 09 22:37:37.719952 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.719792 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wb8tm\"" Jul 09 22:37:37.719952 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.719791 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 22:37:37.719952 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.719913 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 22:37:37.719952 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.719931 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 09 22:37:37.722425 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.722254 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 22:37:37.722425 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.722317 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 22:37:37.722605 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.722583 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-drsgl\"" Jul 09 22:37:37.722934 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.722898 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 22:37:37.724223 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.724205 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr"] Jul 09 22:37:37.760092 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.760055 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:37.785060 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.785023 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ccfb6a57-528d-4548-9b47-571f9b97f80a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-c6jvr\" (UID: \"ccfb6a57-528d-4548-9b47-571f9b97f80a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:37.785060 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.785063 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9lw4\" (UniqueName: \"kubernetes.io/projected/650070d8-38ef-4bd2-a9de-6eee8e5fff60-kube-api-access-h9lw4\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.785310 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.785089 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccfb6a57-528d-4548-9b47-571f9b97f80a-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-c6jvr\" (UID: \"ccfb6a57-528d-4548-9b47-571f9b97f80a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:37.785310 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.785116 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/650070d8-38ef-4bd2-a9de-6eee8e5fff60-node-exporter-accelerators-collector-config\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.785310 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.785139 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ccfb6a57-528d-4548-9b47-571f9b97f80a-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-c6jvr\" (UID: \"ccfb6a57-528d-4548-9b47-571f9b97f80a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:37.785310 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.785156 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccfb6a57-528d-4548-9b47-571f9b97f80a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-c6jvr\" (UID: \"ccfb6a57-528d-4548-9b47-571f9b97f80a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:37.785310 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.785216 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/650070d8-38ef-4bd2-a9de-6eee8e5fff60-node-exporter-wtmp\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.785310 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.785261 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfvxh\" (UniqueName: \"kubernetes.io/projected/ccfb6a57-528d-4548-9b47-571f9b97f80a-kube-api-access-zfvxh\") pod \"kube-state-metrics-7c9dbb8b7d-c6jvr\" (UID: \"ccfb6a57-528d-4548-9b47-571f9b97f80a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:37.785596 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.785324 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/650070d8-38ef-4bd2-a9de-6eee8e5fff60-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.785596 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.785380 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/650070d8-38ef-4bd2-a9de-6eee8e5fff60-root\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.785596 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.785407 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/650070d8-38ef-4bd2-a9de-6eee8e5fff60-node-exporter-textfile\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.785596 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.785431 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/650070d8-38ef-4bd2-a9de-6eee8e5fff60-node-exporter-tls\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.785596 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.785467 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/650070d8-38ef-4bd2-a9de-6eee8e5fff60-metrics-client-ca\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.785596 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.785526 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/650070d8-38ef-4bd2-a9de-6eee8e5fff60-sys\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.785596 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.785550 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccfb6a57-528d-4548-9b47-571f9b97f80a-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-c6jvr\" (UID: \"ccfb6a57-528d-4548-9b47-571f9b97f80a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:37.886218 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886182 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/650070d8-38ef-4bd2-a9de-6eee8e5fff60-root\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.886218 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886222 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/650070d8-38ef-4bd2-a9de-6eee8e5fff60-node-exporter-textfile\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.886447 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886241 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/650070d8-38ef-4bd2-a9de-6eee8e5fff60-node-exporter-tls\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.886447 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886259 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/650070d8-38ef-4bd2-a9de-6eee8e5fff60-metrics-client-ca\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.886447 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886282 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/650070d8-38ef-4bd2-a9de-6eee8e5fff60-sys\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.886447 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886296 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/650070d8-38ef-4bd2-a9de-6eee8e5fff60-root\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.886447 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886309 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccfb6a57-528d-4548-9b47-571f9b97f80a-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-c6jvr\" (UID: \"ccfb6a57-528d-4548-9b47-571f9b97f80a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:37.886447 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886337 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ccfb6a57-528d-4548-9b47-571f9b97f80a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-c6jvr\" (UID: \"ccfb6a57-528d-4548-9b47-571f9b97f80a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:37.886447 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:37.886376 2568 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 09 22:37:37.886447 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:37.886400 2568 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 09 22:37:37.886447 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886373 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/650070d8-38ef-4bd2-a9de-6eee8e5fff60-sys\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.886879 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:37.886450 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/650070d8-38ef-4bd2-a9de-6eee8e5fff60-node-exporter-tls podName:650070d8-38ef-4bd2-a9de-6eee8e5fff60 nodeName:}" failed. No retries permitted until 2026-07-09 22:37:38.386427923 +0000 UTC m=+282.927506704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/650070d8-38ef-4bd2-a9de-6eee8e5fff60-node-exporter-tls") pod "node-exporter-dw4vg" (UID: "650070d8-38ef-4bd2-a9de-6eee8e5fff60") : secret "node-exporter-tls" not found Jul 09 22:37:37.886879 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:37.886509 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccfb6a57-528d-4548-9b47-571f9b97f80a-kube-state-metrics-tls podName:ccfb6a57-528d-4548-9b47-571f9b97f80a nodeName:}" failed. No retries permitted until 2026-07-09 22:37:38.386469262 +0000 UTC m=+282.927548037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ccfb6a57-528d-4548-9b47-571f9b97f80a-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-c6jvr" (UID: "ccfb6a57-528d-4548-9b47-571f9b97f80a") : secret "kube-state-metrics-tls" not found Jul 09 22:37:37.886879 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886529 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9lw4\" (UniqueName: \"kubernetes.io/projected/650070d8-38ef-4bd2-a9de-6eee8e5fff60-kube-api-access-h9lw4\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.886879 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886566 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccfb6a57-528d-4548-9b47-571f9b97f80a-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-c6jvr\" (UID: \"ccfb6a57-528d-4548-9b47-571f9b97f80a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:37.886879 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886597 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/650070d8-38ef-4bd2-a9de-6eee8e5fff60-node-exporter-accelerators-collector-config\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.886879 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886620 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ccfb6a57-528d-4548-9b47-571f9b97f80a-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-c6jvr\" (UID: \"ccfb6a57-528d-4548-9b47-571f9b97f80a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:37.886879 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886634 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/650070d8-38ef-4bd2-a9de-6eee8e5fff60-node-exporter-textfile\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.886879 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886650 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccfb6a57-528d-4548-9b47-571f9b97f80a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-c6jvr\" (UID: \"ccfb6a57-528d-4548-9b47-571f9b97f80a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:37.886879 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886687 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/650070d8-38ef-4bd2-a9de-6eee8e5fff60-node-exporter-wtmp\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.886879 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886715 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvxh\" (UniqueName: \"kubernetes.io/projected/ccfb6a57-528d-4548-9b47-571f9b97f80a-kube-api-access-zfvxh\") pod \"kube-state-metrics-7c9dbb8b7d-c6jvr\" (UID: \"ccfb6a57-528d-4548-9b47-571f9b97f80a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:37.886879 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886779 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/650070d8-38ef-4bd2-a9de-6eee8e5fff60-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.886879 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886838 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/650070d8-38ef-4bd2-a9de-6eee8e5fff60-node-exporter-wtmp\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.887309 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886926 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/650070d8-38ef-4bd2-a9de-6eee8e5fff60-metrics-client-ca\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.887309 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.886966 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ccfb6a57-528d-4548-9b47-571f9b97f80a-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-c6jvr\" (UID: \"ccfb6a57-528d-4548-9b47-571f9b97f80a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:37.887309 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.887091 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ccfb6a57-528d-4548-9b47-571f9b97f80a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-c6jvr\" (UID: \"ccfb6a57-528d-4548-9b47-571f9b97f80a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:37.887309 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.887122 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/650070d8-38ef-4bd2-a9de-6eee8e5fff60-node-exporter-accelerators-collector-config\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.887539 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.887521 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccfb6a57-528d-4548-9b47-571f9b97f80a-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-c6jvr\" (UID: \"ccfb6a57-528d-4548-9b47-571f9b97f80a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:37.889137 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.889112 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/650070d8-38ef-4bd2-a9de-6eee8e5fff60-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.889294 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.889273 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccfb6a57-528d-4548-9b47-571f9b97f80a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-c6jvr\" (UID: \"ccfb6a57-528d-4548-9b47-571f9b97f80a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:37.897460 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.897429 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfvxh\" (UniqueName: \"kubernetes.io/projected/ccfb6a57-528d-4548-9b47-571f9b97f80a-kube-api-access-zfvxh\") pod \"kube-state-metrics-7c9dbb8b7d-c6jvr\" (UID: \"ccfb6a57-528d-4548-9b47-571f9b97f80a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:37.897981 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.897965 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9lw4\" (UniqueName: \"kubernetes.io/projected/650070d8-38ef-4bd2-a9de-6eee8e5fff60-kube-api-access-h9lw4\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:37.928456 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:37.928418 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:38.035652 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.035564 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:38.130025 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.129986 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:38.231048 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.231015 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:38.331119 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.331084 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:38.390234 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.390203 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccfb6a57-528d-4548-9b47-571f9b97f80a-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-c6jvr\" (UID: \"ccfb6a57-528d-4548-9b47-571f9b97f80a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:38.390414 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.390393 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/650070d8-38ef-4bd2-a9de-6eee8e5fff60-node-exporter-tls\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:38.392577 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.392539 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/650070d8-38ef-4bd2-a9de-6eee8e5fff60-node-exporter-tls\") pod \"node-exporter-dw4vg\" (UID: \"650070d8-38ef-4bd2-a9de-6eee8e5fff60\") " pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:38.392691 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.392657 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccfb6a57-528d-4548-9b47-571f9b97f80a-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-c6jvr\" (UID: \"ccfb6a57-528d-4548-9b47-571f9b97f80a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:38.600586 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:38.600461 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mzf9t" podUID="e818597c-2b00-4cd2-9d5a-9af19eca0605" Jul 09 22:37:38.626053 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.626022 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" Jul 09 22:37:38.631693 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.631662 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dw4vg" Jul 09 22:37:38.643159 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:37:38.643129 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod650070d8_38ef_4bd2_a9de_6eee8e5fff60.slice/crio-7ab53d1ead6e62ccef711e571af2b4abb17081522456b45a0f1df96416cfed33 WatchSource:0}: Error finding container 7ab53d1ead6e62ccef711e571af2b4abb17081522456b45a0f1df96416cfed33: Status 404 returned error can't find the container with id 7ab53d1ead6e62ccef711e571af2b4abb17081522456b45a0f1df96416cfed33 Jul 09 22:37:38.674434 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.674397 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:38.768059 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.767419 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr"] Jul 09 22:37:38.770664 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.770614 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:38.771253 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.771221 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 22:37:38.773335 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:37:38.773311 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccfb6a57_528d_4548_9b47_571f9b97f80a.slice/crio-5f7afef6355b43682217d58242af71d530220f46d48e9e532b5ce26c2ae403c5 WatchSource:0}: Error finding container 5f7afef6355b43682217d58242af71d530220f46d48e9e532b5ce26c2ae403c5: Status 404 returned error can't find the container with id 5f7afef6355b43682217d58242af71d530220f46d48e9e532b5ce26c2ae403c5 Jul 09 22:37:38.776502 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.776472 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.779619 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.779600 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 09 22:37:38.779723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.779602 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 09 22:37:38.779723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.779603 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 09 22:37:38.779723 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.779603 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 09 22:37:38.779987 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.779948 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 09 22:37:38.780108 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.780022 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 09 22:37:38.780108 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.780061 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 09 22:37:38.780246 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.780107 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 09 22:37:38.780246 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.780174 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 09 22:37:38.780332 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.780271 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mhj2k\"" Jul 09 22:37:38.791435 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.791410 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 22:37:38.894301 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.894213 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.894301 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.894271 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.894562 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.894345 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.894562 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.894364 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-config-out\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.894562 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.894382 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.894562 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.894408 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.894562 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.894461 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.894747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.894581 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.894747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.894607 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-config-volume\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.894747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.894622 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.894747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.894639 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.894747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.894662 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-web-config\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.894747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.894677 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcpd2\" (UniqueName: \"kubernetes.io/projected/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-kube-api-access-bcpd2\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.964262 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.964221 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dw4vg" event={"ID":"650070d8-38ef-4bd2-a9de-6eee8e5fff60","Type":"ContainerStarted","Data":"7ab53d1ead6e62ccef711e571af2b4abb17081522456b45a0f1df96416cfed33"} Jul 09 22:37:38.965664 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.965635 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mzf9t" Jul 09 22:37:38.965789 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.965639 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" event={"ID":"ccfb6a57-528d-4548-9b47-571f9b97f80a","Type":"ContainerStarted","Data":"5f7afef6355b43682217d58242af71d530220f46d48e9e532b5ce26c2ae403c5"} Jul 09 22:37:38.995509 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.995452 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.995647 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.995522 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-config-out\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.995647 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.995551 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.995647 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.995586 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.995647 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.995608 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.995647 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:38.995614 2568 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 09 22:37:38.995647 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.995642 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.995936 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:38.995677 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-main-tls podName:ddab4fd6-05c7-41a3-b581-3ff08941fdaa nodeName:}" failed. No retries permitted until 2026-07-09 22:37:39.495656677 +0000 UTC m=+284.036735460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "ddab4fd6-05c7-41a3-b581-3ff08941fdaa") : secret "alertmanager-main-tls" not found Jul 09 22:37:38.995936 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.995710 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-config-volume\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.995936 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.995733 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.995936 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.995758 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.995936 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.995783 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-web-config\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.995936 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.995806 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcpd2\" (UniqueName: \"kubernetes.io/projected/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-kube-api-access-bcpd2\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.995936 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.995833 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.995936 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.995872 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.996370 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.996052 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.996990 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.996943 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.997319 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.997293 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.998644 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.998622 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-config-out\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.998943 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.998925 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.999016 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.998987 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.999146 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.999100 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.999146 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.999118 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-config-volume\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.999466 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.999447 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.999898 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.999876 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:38.999998 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:38.999982 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-web-config\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:39.004758 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:39.004734 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcpd2\" (UniqueName: \"kubernetes.io/projected/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-kube-api-access-bcpd2\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:39.154715 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:39.154626 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:39.256852 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:39.256814 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:39.499982 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:39.499897 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:39.503125 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:39.503100 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:39.686529 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:39.686483 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:37:39.852815 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:39.852797 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 22:37:40.097818 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:37:40.097784 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddab4fd6_05c7_41a3_b581_3ff08941fdaa.slice/crio-1a70bbfbbe258bcc4814bdea500e397c45e1c082db1fc3b91e20e31c6c856f8e WatchSource:0}: Error finding container 1a70bbfbbe258bcc4814bdea500e397c45e1c082db1fc3b91e20e31c6c856f8e: Status 404 returned error can't find the container with id 1a70bbfbbe258bcc4814bdea500e397c45e1c082db1fc3b91e20e31c6c856f8e Jul 09 22:37:40.569631 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:40.569587 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:40.667412 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:40.667237 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:40.733016 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:40.732962 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:40.897675 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:40.897572 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:40.973273 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:40.973237 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddab4fd6-05c7-41a3-b581-3ff08941fdaa","Type":"ContainerStarted","Data":"1a70bbfbbe258bcc4814bdea500e397c45e1c082db1fc3b91e20e31c6c856f8e"} Jul 09 22:37:40.974848 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:40.974816 2568 generic.go:356] "Generic (PLEG): container finished" podID="650070d8-38ef-4bd2-a9de-6eee8e5fff60" containerID="70b655ad4d1d6d7f0f14e4362c7fc45dad8e392867c4405fc9de5fa48044ad9f" exitCode=0 Jul 09 22:37:40.974975 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:40.974892 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dw4vg" event={"ID":"650070d8-38ef-4bd2-a9de-6eee8e5fff60","Type":"ContainerDied","Data":"70b655ad4d1d6d7f0f14e4362c7fc45dad8e392867c4405fc9de5fa48044ad9f"} Jul 09 22:37:40.977066 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:40.977043 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" event={"ID":"ccfb6a57-528d-4548-9b47-571f9b97f80a","Type":"ContainerStarted","Data":"a33b9aae4724b1704baa958b5865f5867420edf704240b84c636c275c428917b"} Jul 09 22:37:40.977164 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:40.977087 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" event={"ID":"ccfb6a57-528d-4548-9b47-571f9b97f80a","Type":"ContainerStarted","Data":"b92c34788d0c11ede5c95e586a1a72652ad68a24be52fd3d27f339fff4c6d835"} Jul 09 22:37:40.977164 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:40.977106 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" event={"ID":"ccfb6a57-528d-4548-9b47-571f9b97f80a","Type":"ContainerStarted","Data":"79ccde555480046563e9f1d32195bad0bc54f936c299d324d45bc10bcfc07d78"} Jul 09 22:37:40.998185 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:40.998132 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:41.014860 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:41.014801 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-c6jvr" podStartSLOduration=2.422990115 podStartE2EDuration="4.014781408s" totalImagesPullingTime="1.591791293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:37:41.013641601 +0000 UTC m=+285.554720408" watchObservedRunningTime="2026-07-09 22:37:41.014781408 +0000 UTC m=+285.555860201" Jul 09 22:37:41.823540 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:41.823482 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:37:41.825740 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:41.825718 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e818597c-2b00-4cd2-9d5a-9af19eca0605-metrics-tls\") pod \"dns-default-mzf9t\" (UID: \"e818597c-2b00-4cd2-9d5a-9af19eca0605\") " pod="openshift-dns/dns-default-mzf9t" Jul 09 22:37:41.970069 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:41.969992 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xdqcx\"" Jul 09 22:37:41.977986 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:41.977956 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mzf9t" Jul 09 22:37:41.982223 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:41.982195 2568 generic.go:356] "Generic (PLEG): container finished" podID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerID="c3bfbf17dd015c88cdef28788f8f2a1d7346e8c2bd2140c38faffcbff2e4a797" exitCode=0 Jul 09 22:37:41.982355 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:41.982286 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddab4fd6-05c7-41a3-b581-3ff08941fdaa","Type":"ContainerDied","Data":"c3bfbf17dd015c88cdef28788f8f2a1d7346e8c2bd2140c38faffcbff2e4a797"} Jul 09 22:37:41.984443 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:41.984420 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dw4vg" event={"ID":"650070d8-38ef-4bd2-a9de-6eee8e5fff60","Type":"ContainerStarted","Data":"baaa2816f18d0b80db968f72640c5fe1858b90f7a57da2f2e72d95a2b5e8588f"} Jul 09 22:37:41.984521 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:41.984503 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dw4vg" event={"ID":"650070d8-38ef-4bd2-a9de-6eee8e5fff60","Type":"ContainerStarted","Data":"12a99ea14a63009e16d07758638c9afb287f168e042b363099b5b1bfad778238"} Jul 09 22:37:42.029361 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:42.029301 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dw4vg" podStartSLOduration=3.213833907 podStartE2EDuration="5.029285431s" totalImagesPullingTime="1.724563516s" totalInitContainerRuntime="90.888008ms" isStatefulPod=true podCreationTimestamp="2026-07-09 22:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:37:42.0281373 +0000 UTC m=+286.569216120" watchObservedRunningTime="2026-07-09 22:37:42.029285431 +0000 UTC m=+286.570364267" Jul 09 22:37:42.147852 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:42.147824 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mzf9t"] Jul 09 22:37:42.151334 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:37:42.151309 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode818597c_2b00_4cd2_9d5a_9af19eca0605.slice/crio-08303c3009b372e2c42d3332b5c35afcd38b025758be70376d0bcd8d7eff8825 WatchSource:0}: Error finding container 08303c3009b372e2c42d3332b5c35afcd38b025758be70376d0bcd8d7eff8825: Status 404 returned error can't find the container with id 08303c3009b372e2c42d3332b5c35afcd38b025758be70376d0bcd8d7eff8825 Jul 09 22:37:42.479097 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:42.479065 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:42.571762 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:42.571730 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:42.989411 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:42.989380 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mzf9t" event={"ID":"e818597c-2b00-4cd2-9d5a-9af19eca0605","Type":"ContainerStarted","Data":"08303c3009b372e2c42d3332b5c35afcd38b025758be70376d0bcd8d7eff8825"} Jul 09 22:37:43.270870 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:43.270780 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:43.334568 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:43.334529 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:43.714781 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:43.714693 2568 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" podUID="12015fd4-71af-4354-b45f-fb871e9e5e05" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 09 22:37:43.994179 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:43.994083 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:44.099965 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:44.099917 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:44.162733 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:44.162702 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:44.271023 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:44.270943 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:44.329935 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:44.329898 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:45.000156 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:45.000120 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddab4fd6-05c7-41a3-b581-3ff08941fdaa","Type":"ContainerStarted","Data":"d5ea136bfeb1f55460396079096f00751cbbffcc334b9af2b4f2104424672478"} Jul 09 22:37:45.000596 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:45.000217 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddab4fd6-05c7-41a3-b581-3ff08941fdaa","Type":"ContainerStarted","Data":"55ee17369c5992e78344a4929395ef7d170493859125e16013502d7eac52e852"} Jul 09 22:37:45.000596 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:45.000248 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddab4fd6-05c7-41a3-b581-3ff08941fdaa","Type":"ContainerStarted","Data":"71a6ecb1a1579358a9f97d9b4efaa10d4a504b08f7f09e355915ae19a1b25904"} Jul 09 22:37:45.000596 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:45.000275 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddab4fd6-05c7-41a3-b581-3ff08941fdaa","Type":"ContainerStarted","Data":"bb0492fa481b3463e1aa9b83dc8c247971aa91a6df02222934c6c2c2e81201ab"} Jul 09 22:37:45.000596 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:45.000309 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddab4fd6-05c7-41a3-b581-3ff08941fdaa","Type":"ContainerStarted","Data":"5d2945d02dd033c41ba8349c30e4274bc5954e452b1f78ec647f1bf6165cc788"} Jul 09 22:37:45.001658 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:45.001638 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mzf9t" event={"ID":"e818597c-2b00-4cd2-9d5a-9af19eca0605","Type":"ContainerStarted","Data":"975515bdaec3f8439905c2b1f0adf7e0ae3db2d346fb412fbb991c690d64c9bb"} Jul 09 22:37:45.001771 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:45.001674 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mzf9t" event={"ID":"e818597c-2b00-4cd2-9d5a-9af19eca0605","Type":"ContainerStarted","Data":"97ba4a4195bdf4dac39d42ec3dbc0791d0465effdb9dcc0ac43b888249285e32"} Jul 09 22:37:45.001839 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:45.001778 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mzf9t" Jul 09 22:37:45.023891 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:45.023846 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mzf9t" podStartSLOduration=251.826586959 podStartE2EDuration="4m14.023831147s" totalImagesPullingTime="2.197244188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:33:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:37:45.022216767 +0000 UTC m=+289.563295569" watchObservedRunningTime="2026-07-09 22:37:45.023831147 +0000 UTC m=+289.564909957" Jul 09 22:37:45.131061 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:45.131028 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:45.197380 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:45.197338 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:45.299057 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:45.298967 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:46.008853 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.008818 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddab4fd6-05c7-41a3-b581-3ff08941fdaa","Type":"ContainerStarted","Data":"52cb2eff9019a0561c14b7b4ae99674801ac98e3d45ca2910040f7dbdaf6dccc"} Jul 09 22:37:46.042051 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.041994 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.29465574 podStartE2EDuration="8.041972753s" totalImagesPullingTime="4.726388913s" totalInitContainerRuntime="20.9281ms" isStatefulPod=false podCreationTimestamp="2026-07-09 22:37:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:37:46.039901219 +0000 UTC m=+290.580980013" watchObservedRunningTime="2026-07-09 22:37:46.041972753 +0000 UTC m=+290.583051546" Jul 09 22:37:46.207642 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.207608 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:46.263681 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.263601 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:46.596601 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.596561 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-db664c774-kspt4" podUID="ab92fa47-2296-41b9-9636-b98ffd793b8a" containerName="registry" containerID="cri-o://89d6f34b440478bb65e7ee7b168ab71229e3bdb477074acdcdf90173c225141c" gracePeriod=30 Jul 09 22:37:46.838667 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.838629 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:37:46.969082 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.968974 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab92fa47-2296-41b9-9636-b98ffd793b8a-installation-pull-secrets\") pod \"ab92fa47-2296-41b9-9636-b98ffd793b8a\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " Jul 09 22:37:46.969082 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.969041 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls\") pod \"ab92fa47-2296-41b9-9636-b98ffd793b8a\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " Jul 09 22:37:46.969082 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.969072 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-certificates\") pod \"ab92fa47-2296-41b9-9636-b98ffd793b8a\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " Jul 09 22:37:46.969346 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.969092 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4pxt\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-kube-api-access-t4pxt\") pod \"ab92fa47-2296-41b9-9636-b98ffd793b8a\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " Jul 09 22:37:46.969346 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.969111 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab92fa47-2296-41b9-9636-b98ffd793b8a-trusted-ca\") pod \"ab92fa47-2296-41b9-9636-b98ffd793b8a\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " Jul 09 22:37:46.969346 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.969137 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab92fa47-2296-41b9-9636-b98ffd793b8a-image-registry-private-configuration\") pod \"ab92fa47-2296-41b9-9636-b98ffd793b8a\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " Jul 09 22:37:46.969346 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.969159 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab92fa47-2296-41b9-9636-b98ffd793b8a-ca-trust-extracted\") pod \"ab92fa47-2296-41b9-9636-b98ffd793b8a\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " Jul 09 22:37:46.969567 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.969482 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-bound-sa-token\") pod \"ab92fa47-2296-41b9-9636-b98ffd793b8a\" (UID: \"ab92fa47-2296-41b9-9636-b98ffd793b8a\") " Jul 09 22:37:46.969621 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.969568 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ab92fa47-2296-41b9-9636-b98ffd793b8a" (UID: "ab92fa47-2296-41b9-9636-b98ffd793b8a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:37:46.969708 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.969659 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab92fa47-2296-41b9-9636-b98ffd793b8a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ab92fa47-2296-41b9-9636-b98ffd793b8a" (UID: "ab92fa47-2296-41b9-9636-b98ffd793b8a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:37:46.969879 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.969825 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab92fa47-2296-41b9-9636-b98ffd793b8a-trusted-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:37:46.969879 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.969849 2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-certificates\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:37:46.971479 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.971424 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ab92fa47-2296-41b9-9636-b98ffd793b8a" (UID: "ab92fa47-2296-41b9-9636-b98ffd793b8a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:37:46.971681 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.971561 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-kube-api-access-t4pxt" (OuterVolumeSpecName: "kube-api-access-t4pxt") pod "ab92fa47-2296-41b9-9636-b98ffd793b8a" (UID: "ab92fa47-2296-41b9-9636-b98ffd793b8a"). InnerVolumeSpecName "kube-api-access-t4pxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:37:46.971778 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.971759 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab92fa47-2296-41b9-9636-b98ffd793b8a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ab92fa47-2296-41b9-9636-b98ffd793b8a" (UID: "ab92fa47-2296-41b9-9636-b98ffd793b8a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:37:46.971855 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.971837 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab92fa47-2296-41b9-9636-b98ffd793b8a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ab92fa47-2296-41b9-9636-b98ffd793b8a" (UID: "ab92fa47-2296-41b9-9636-b98ffd793b8a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:37:46.972076 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.972056 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ab92fa47-2296-41b9-9636-b98ffd793b8a" (UID: "ab92fa47-2296-41b9-9636-b98ffd793b8a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:37:46.977604 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:46.977577 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab92fa47-2296-41b9-9636-b98ffd793b8a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ab92fa47-2296-41b9-9636-b98ffd793b8a" (UID: "ab92fa47-2296-41b9-9636-b98ffd793b8a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:37:47.013342 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:47.013305 2568 generic.go:356] "Generic (PLEG): container finished" podID="ab92fa47-2296-41b9-9636-b98ffd793b8a" containerID="89d6f34b440478bb65e7ee7b168ab71229e3bdb477074acdcdf90173c225141c" exitCode=0 Jul 09 22:37:47.013820 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:47.013362 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-db664c774-kspt4" Jul 09 22:37:47.013820 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:47.013390 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-db664c774-kspt4" event={"ID":"ab92fa47-2296-41b9-9636-b98ffd793b8a","Type":"ContainerDied","Data":"89d6f34b440478bb65e7ee7b168ab71229e3bdb477074acdcdf90173c225141c"} Jul 09 22:37:47.013820 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:47.013461 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-db664c774-kspt4" event={"ID":"ab92fa47-2296-41b9-9636-b98ffd793b8a","Type":"ContainerDied","Data":"a170c3ecdfb65b83935e67251f4b66eaee86b29c02360162c6dfac5ebd6fc449"} Jul 09 22:37:47.013820 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:47.013481 2568 scope.go:117] "RemoveContainer" containerID="89d6f34b440478bb65e7ee7b168ab71229e3bdb477074acdcdf90173c225141c" Jul 09 22:37:47.021877 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:47.021860 2568 scope.go:117] "RemoveContainer" containerID="89d6f34b440478bb65e7ee7b168ab71229e3bdb477074acdcdf90173c225141c" Jul 09 22:37:47.022122 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:47.022097 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d6f34b440478bb65e7ee7b168ab71229e3bdb477074acdcdf90173c225141c\": container with ID starting with 89d6f34b440478bb65e7ee7b168ab71229e3bdb477074acdcdf90173c225141c not found: ID does not exist" containerID="89d6f34b440478bb65e7ee7b168ab71229e3bdb477074acdcdf90173c225141c" Jul 09 22:37:47.022161 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:47.022131 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d6f34b440478bb65e7ee7b168ab71229e3bdb477074acdcdf90173c225141c"} err="failed to get container status \"89d6f34b440478bb65e7ee7b168ab71229e3bdb477074acdcdf90173c225141c\": rpc error: code = NotFound desc = could not find container \"89d6f34b440478bb65e7ee7b168ab71229e3bdb477074acdcdf90173c225141c\": container with ID starting with 89d6f34b440478bb65e7ee7b168ab71229e3bdb477074acdcdf90173c225141c not found: ID does not exist" Jul 09 22:37:47.038184 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:47.038145 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-db664c774-kspt4"] Jul 09 22:37:47.046920 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:47.046894 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-db664c774-kspt4"] Jul 09 22:37:47.071230 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:47.071197 2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab92fa47-2296-41b9-9636-b98ffd793b8a-installation-pull-secrets\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:37:47.071230 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:47.071222 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-registry-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:37:47.071230 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:47.071232 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4pxt\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-kube-api-access-t4pxt\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:37:47.071446 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:47.071242 2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab92fa47-2296-41b9-9636-b98ffd793b8a-image-registry-private-configuration\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:37:47.071446 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:47.071252 2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab92fa47-2296-41b9-9636-b98ffd793b8a-ca-trust-extracted\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:37:47.071446 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:47.071260 2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab92fa47-2296-41b9-9636-b98ffd793b8a-bound-sa-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:37:48.046273 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:48.046235 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab92fa47-2296-41b9-9636-b98ffd793b8a" path="/var/lib/kubelet/pods/ab92fa47-2296-41b9-9636-b98ffd793b8a/volumes" Jul 09 22:37:51.351112 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:51.351075 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:51.450802 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:51.450775 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:53.042886 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:37:53.042846 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-5wch9" podUID="941cb948-d902-4c71-abf2-52ec25863dd8" Jul 09 22:37:53.340018 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:53.339993 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:53.442224 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:53.442192 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:53.526300 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:53.526258 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:53.714273 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:53.714184 2568 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" podUID="12015fd4-71af-4354-b45f-fb871e9e5e05" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 09 22:37:55.011478 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:55.011453 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mzf9t" Jul 09 22:37:55.879340 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:55.879313 2568 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 22:37:56.120847 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:56.120753 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:56.185781 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:56.185747 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:58.304191 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:58.304160 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:37:58.404256 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:37:58.404223 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:38:03.714647 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:03.714561 2568 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" podUID="12015fd4-71af-4354-b45f-fb871e9e5e05" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 09 22:38:03.714986 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:03.714655 2568 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" Jul 09 22:38:03.715119 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:03.715102 2568 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"4c83ffb37c58a15bf78abb4b70593bd3942f83206cf420a75dba4d2b5150d1bb"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 09 22:38:03.715155 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:03.715139 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" podUID="12015fd4-71af-4354-b45f-fb871e9e5e05" containerName="service-proxy" containerID="cri-o://4c83ffb37c58a15bf78abb4b70593bd3942f83206cf420a75dba4d2b5150d1bb" gracePeriod=30 Jul 09 22:38:04.073323 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:04.073291 2568 generic.go:356] "Generic (PLEG): container finished" podID="12015fd4-71af-4354-b45f-fb871e9e5e05" containerID="4c83ffb37c58a15bf78abb4b70593bd3942f83206cf420a75dba4d2b5150d1bb" exitCode=2 Jul 09 22:38:04.073522 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:04.073363 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" event={"ID":"12015fd4-71af-4354-b45f-fb871e9e5e05","Type":"ContainerDied","Data":"4c83ffb37c58a15bf78abb4b70593bd3942f83206cf420a75dba4d2b5150d1bb"} Jul 09 22:38:04.073522 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:04.073404 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7544dfbf9b-ch5mr" event={"ID":"12015fd4-71af-4354-b45f-fb871e9e5e05","Type":"ContainerStarted","Data":"fb9f56ea43aaa7096807ad3e74a7a97f889bae41bf2e9afad7187f5e95061def"} Jul 09 22:38:04.126624 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:04.126596 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:38:04.179446 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:04.179418 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:38:07.042231 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:07.042175 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:38:13.307431 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:13.307399 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:38:13.357104 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:13.357075 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:38:28.296561 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:28.296520 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:38:28.363102 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:28.363076 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:38:39.084727 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:39.084691 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:38:40.503567 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:40.503531 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:38:40.573397 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:40.573363 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:38:55.356703 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:55.356622 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:38:55.420919 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:55.420888 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:38:56.974995 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:56.974961 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:38:57.353279 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:57.353247 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:38:58.012651 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:58.012619 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 22:38:58.013068 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:58.012909 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="alertmanager" containerID="cri-o://5d2945d02dd033c41ba8349c30e4274bc5954e452b1f78ec647f1bf6165cc788" gracePeriod=120 Jul 09 22:38:58.013068 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:58.012983 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="kube-rbac-proxy-web" containerID="cri-o://71a6ecb1a1579358a9f97d9b4efaa10d4a504b08f7f09e355915ae19a1b25904" gracePeriod=120 Jul 09 22:38:58.013068 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:58.012987 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="kube-rbac-proxy-metric" containerID="cri-o://d5ea136bfeb1f55460396079096f00751cbbffcc334b9af2b4f2104424672478" gracePeriod=120 Jul 09 22:38:58.013068 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:58.013032 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="kube-rbac-proxy" containerID="cri-o://55ee17369c5992e78344a4929395ef7d170493859125e16013502d7eac52e852" gracePeriod=120 Jul 09 22:38:58.013289 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:58.013077 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="prom-label-proxy" containerID="cri-o://52cb2eff9019a0561c14b7b4ae99674801ac98e3d45ca2910040f7dbdaf6dccc" gracePeriod=120 Jul 09 22:38:58.013289 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:58.013043 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="config-reloader" containerID="cri-o://bb0492fa481b3463e1aa9b83dc8c247971aa91a6df02222934c6c2c2e81201ab" gracePeriod=120 Jul 09 22:38:58.263426 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:58.263345 2568 generic.go:356] "Generic (PLEG): container finished" podID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerID="52cb2eff9019a0561c14b7b4ae99674801ac98e3d45ca2910040f7dbdaf6dccc" exitCode=0 Jul 09 22:38:58.263426 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:58.263369 2568 generic.go:356] "Generic (PLEG): container finished" podID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerID="55ee17369c5992e78344a4929395ef7d170493859125e16013502d7eac52e852" exitCode=0 Jul 09 22:38:58.263426 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:58.263375 2568 generic.go:356] "Generic (PLEG): container finished" podID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerID="bb0492fa481b3463e1aa9b83dc8c247971aa91a6df02222934c6c2c2e81201ab" exitCode=0 Jul 09 22:38:58.263426 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:58.263381 2568 generic.go:356] "Generic (PLEG): container finished" podID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerID="5d2945d02dd033c41ba8349c30e4274bc5954e452b1f78ec647f1bf6165cc788" exitCode=0 Jul 09 22:38:58.263680 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:58.263418 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddab4fd6-05c7-41a3-b581-3ff08941fdaa","Type":"ContainerDied","Data":"52cb2eff9019a0561c14b7b4ae99674801ac98e3d45ca2910040f7dbdaf6dccc"} Jul 09 22:38:58.263680 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:58.263506 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddab4fd6-05c7-41a3-b581-3ff08941fdaa","Type":"ContainerDied","Data":"55ee17369c5992e78344a4929395ef7d170493859125e16013502d7eac52e852"} Jul 09 22:38:58.263680 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:58.263541 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddab4fd6-05c7-41a3-b581-3ff08941fdaa","Type":"ContainerDied","Data":"bb0492fa481b3463e1aa9b83dc8c247971aa91a6df02222934c6c2c2e81201ab"} Jul 09 22:38:58.263680 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:58.263566 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddab4fd6-05c7-41a3-b581-3ff08941fdaa","Type":"ContainerDied","Data":"5d2945d02dd033c41ba8349c30e4274bc5954e452b1f78ec647f1bf6165cc788"} Jul 09 22:38:59.250231 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.250204 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.270032 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.269997 2568 generic.go:356] "Generic (PLEG): container finished" podID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerID="d5ea136bfeb1f55460396079096f00751cbbffcc334b9af2b4f2104424672478" exitCode=0 Jul 09 22:38:59.270032 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.270022 2568 generic.go:356] "Generic (PLEG): container finished" podID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerID="71a6ecb1a1579358a9f97d9b4efaa10d4a504b08f7f09e355915ae19a1b25904" exitCode=0 Jul 09 22:38:59.270270 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.270048 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddab4fd6-05c7-41a3-b581-3ff08941fdaa","Type":"ContainerDied","Data":"d5ea136bfeb1f55460396079096f00751cbbffcc334b9af2b4f2104424672478"} Jul 09 22:38:59.270270 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.270076 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.270270 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.270119 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddab4fd6-05c7-41a3-b581-3ff08941fdaa","Type":"ContainerDied","Data":"71a6ecb1a1579358a9f97d9b4efaa10d4a504b08f7f09e355915ae19a1b25904"} Jul 09 22:38:59.270270 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.270159 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ddab4fd6-05c7-41a3-b581-3ff08941fdaa","Type":"ContainerDied","Data":"1a70bbfbbe258bcc4814bdea500e397c45e1c082db1fc3b91e20e31c6c856f8e"} Jul 09 22:38:59.270270 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.270187 2568 scope.go:117] "RemoveContainer" containerID="52cb2eff9019a0561c14b7b4ae99674801ac98e3d45ca2910040f7dbdaf6dccc" Jul 09 22:38:59.277928 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.277903 2568 scope.go:117] "RemoveContainer" containerID="d5ea136bfeb1f55460396079096f00751cbbffcc334b9af2b4f2104424672478" Jul 09 22:38:59.287370 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.287348 2568 scope.go:117] "RemoveContainer" containerID="55ee17369c5992e78344a4929395ef7d170493859125e16013502d7eac52e852" Jul 09 22:38:59.295386 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.295364 2568 scope.go:117] "RemoveContainer" containerID="71a6ecb1a1579358a9f97d9b4efaa10d4a504b08f7f09e355915ae19a1b25904" Jul 09 22:38:59.302375 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.302356 2568 scope.go:117] "RemoveContainer" containerID="bb0492fa481b3463e1aa9b83dc8c247971aa91a6df02222934c6c2c2e81201ab" Jul 09 22:38:59.309675 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.309651 2568 scope.go:117] "RemoveContainer" containerID="5d2945d02dd033c41ba8349c30e4274bc5954e452b1f78ec647f1bf6165cc788" Jul 09 22:38:59.311918 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.311897 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-metrics-client-ca\") pod \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " Jul 09 22:38:59.312010 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.311927 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcpd2\" (UniqueName: \"kubernetes.io/projected/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-kube-api-access-bcpd2\") pod \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " Jul 09 22:38:59.312010 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.311955 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-alertmanager-main-db\") pod \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " Jul 09 22:38:59.312131 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.312112 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-config-volume\") pod \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " Jul 09 22:38:59.312215 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.312187 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-main-tls\") pod \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " Jul 09 22:38:59.312288 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.312244 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-config-out\") pod \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " Jul 09 22:38:59.312288 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.312258 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "ddab4fd6-05c7-41a3-b581-3ff08941fdaa" (UID: "ddab4fd6-05c7-41a3-b581-3ff08941fdaa"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:38:59.312288 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.312278 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-web-config\") pod \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " Jul 09 22:38:59.312458 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.312334 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "ddab4fd6-05c7-41a3-b581-3ff08941fdaa" (UID: "ddab4fd6-05c7-41a3-b581-3ff08941fdaa"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:38:59.312458 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.312341 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-kube-rbac-proxy-web\") pod \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " Jul 09 22:38:59.312458 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.312395 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-alertmanager-trusted-ca-bundle\") pod \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " Jul 09 22:38:59.312458 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.312436 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-cluster-tls-config\") pod \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " Jul 09 22:38:59.313240 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.312516 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-tls-assets\") pod \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " Jul 09 22:38:59.313240 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.312545 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-kube-rbac-proxy\") pod \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " Jul 09 22:38:59.313240 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.312595 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\" (UID: \"ddab4fd6-05c7-41a3-b581-3ff08941fdaa\") " Jul 09 22:38:59.313240 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.312813 2568 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-metrics-client-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:38:59.313240 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.312829 2568 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-alertmanager-main-db\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:38:59.314235 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.314178 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "ddab4fd6-05c7-41a3-b581-3ff08941fdaa" (UID: "ddab4fd6-05c7-41a3-b581-3ff08941fdaa"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:38:59.314891 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.314865 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-config-volume" (OuterVolumeSpecName: "config-volume") pod "ddab4fd6-05c7-41a3-b581-3ff08941fdaa" (UID: "ddab4fd6-05c7-41a3-b581-3ff08941fdaa"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:38:59.315443 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.315415 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "ddab4fd6-05c7-41a3-b581-3ff08941fdaa" (UID: "ddab4fd6-05c7-41a3-b581-3ff08941fdaa"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:38:59.316640 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.316611 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "ddab4fd6-05c7-41a3-b581-3ff08941fdaa" (UID: "ddab4fd6-05c7-41a3-b581-3ff08941fdaa"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:38:59.316742 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.316614 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ddab4fd6-05c7-41a3-b581-3ff08941fdaa" (UID: "ddab4fd6-05c7-41a3-b581-3ff08941fdaa"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:38:59.319249 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.317100 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "ddab4fd6-05c7-41a3-b581-3ff08941fdaa" (UID: "ddab4fd6-05c7-41a3-b581-3ff08941fdaa"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:38:59.319568 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.319437 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "ddab4fd6-05c7-41a3-b581-3ff08941fdaa" (UID: "ddab4fd6-05c7-41a3-b581-3ff08941fdaa"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:38:59.319568 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.319530 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-kube-api-access-bcpd2" (OuterVolumeSpecName: "kube-api-access-bcpd2") pod "ddab4fd6-05c7-41a3-b581-3ff08941fdaa" (UID: "ddab4fd6-05c7-41a3-b581-3ff08941fdaa"). InnerVolumeSpecName "kube-api-access-bcpd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:38:59.321850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.321643 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-config-out" (OuterVolumeSpecName: "config-out") pod "ddab4fd6-05c7-41a3-b581-3ff08941fdaa" (UID: "ddab4fd6-05c7-41a3-b581-3ff08941fdaa"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:38:59.322331 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.322200 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "ddab4fd6-05c7-41a3-b581-3ff08941fdaa" (UID: "ddab4fd6-05c7-41a3-b581-3ff08941fdaa"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:38:59.324534 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.324509 2568 scope.go:117] "RemoveContainer" containerID="c3bfbf17dd015c88cdef28788f8f2a1d7346e8c2bd2140c38faffcbff2e4a797" Jul 09 22:38:59.330094 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.330061 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-web-config" (OuterVolumeSpecName: "web-config") pod "ddab4fd6-05c7-41a3-b581-3ff08941fdaa" (UID: "ddab4fd6-05c7-41a3-b581-3ff08941fdaa"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:38:59.333431 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.333412 2568 scope.go:117] "RemoveContainer" containerID="52cb2eff9019a0561c14b7b4ae99674801ac98e3d45ca2910040f7dbdaf6dccc" Jul 09 22:38:59.333789 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:38:59.333767 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52cb2eff9019a0561c14b7b4ae99674801ac98e3d45ca2910040f7dbdaf6dccc\": container with ID starting with 52cb2eff9019a0561c14b7b4ae99674801ac98e3d45ca2910040f7dbdaf6dccc not found: ID does not exist" containerID="52cb2eff9019a0561c14b7b4ae99674801ac98e3d45ca2910040f7dbdaf6dccc" Jul 09 22:38:59.333862 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.333800 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52cb2eff9019a0561c14b7b4ae99674801ac98e3d45ca2910040f7dbdaf6dccc"} err="failed to get container status \"52cb2eff9019a0561c14b7b4ae99674801ac98e3d45ca2910040f7dbdaf6dccc\": rpc error: code = NotFound desc = could not find container \"52cb2eff9019a0561c14b7b4ae99674801ac98e3d45ca2910040f7dbdaf6dccc\": container with ID starting with 52cb2eff9019a0561c14b7b4ae99674801ac98e3d45ca2910040f7dbdaf6dccc not found: ID does not exist" Jul 09 22:38:59.333862 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.333820 2568 scope.go:117] "RemoveContainer" containerID="d5ea136bfeb1f55460396079096f00751cbbffcc334b9af2b4f2104424672478" Jul 09 22:38:59.334082 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:38:59.334063 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5ea136bfeb1f55460396079096f00751cbbffcc334b9af2b4f2104424672478\": container with ID starting with d5ea136bfeb1f55460396079096f00751cbbffcc334b9af2b4f2104424672478 not found: ID does not exist" containerID="d5ea136bfeb1f55460396079096f00751cbbffcc334b9af2b4f2104424672478" Jul 09 22:38:59.334152 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.334091 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5ea136bfeb1f55460396079096f00751cbbffcc334b9af2b4f2104424672478"} err="failed to get container status \"d5ea136bfeb1f55460396079096f00751cbbffcc334b9af2b4f2104424672478\": rpc error: code = NotFound desc = could not find container \"d5ea136bfeb1f55460396079096f00751cbbffcc334b9af2b4f2104424672478\": container with ID starting with d5ea136bfeb1f55460396079096f00751cbbffcc334b9af2b4f2104424672478 not found: ID does not exist" Jul 09 22:38:59.334152 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.334112 2568 scope.go:117] "RemoveContainer" containerID="55ee17369c5992e78344a4929395ef7d170493859125e16013502d7eac52e852" Jul 09 22:38:59.334343 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:38:59.334322 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55ee17369c5992e78344a4929395ef7d170493859125e16013502d7eac52e852\": container with ID starting with 55ee17369c5992e78344a4929395ef7d170493859125e16013502d7eac52e852 not found: ID does not exist" containerID="55ee17369c5992e78344a4929395ef7d170493859125e16013502d7eac52e852" Jul 09 22:38:59.334464 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.334342 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55ee17369c5992e78344a4929395ef7d170493859125e16013502d7eac52e852"} err="failed to get container status \"55ee17369c5992e78344a4929395ef7d170493859125e16013502d7eac52e852\": rpc error: code = NotFound desc = could not find container \"55ee17369c5992e78344a4929395ef7d170493859125e16013502d7eac52e852\": container with ID starting with 55ee17369c5992e78344a4929395ef7d170493859125e16013502d7eac52e852 not found: ID does not exist" Jul 09 22:38:59.334464 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.334355 2568 scope.go:117] "RemoveContainer" containerID="71a6ecb1a1579358a9f97d9b4efaa10d4a504b08f7f09e355915ae19a1b25904" Jul 09 22:38:59.334702 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:38:59.334686 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71a6ecb1a1579358a9f97d9b4efaa10d4a504b08f7f09e355915ae19a1b25904\": container with ID starting with 71a6ecb1a1579358a9f97d9b4efaa10d4a504b08f7f09e355915ae19a1b25904 not found: ID does not exist" containerID="71a6ecb1a1579358a9f97d9b4efaa10d4a504b08f7f09e355915ae19a1b25904" Jul 09 22:38:59.334753 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.334704 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a6ecb1a1579358a9f97d9b4efaa10d4a504b08f7f09e355915ae19a1b25904"} err="failed to get container status \"71a6ecb1a1579358a9f97d9b4efaa10d4a504b08f7f09e355915ae19a1b25904\": rpc error: code = NotFound desc = could not find container \"71a6ecb1a1579358a9f97d9b4efaa10d4a504b08f7f09e355915ae19a1b25904\": container with ID starting with 71a6ecb1a1579358a9f97d9b4efaa10d4a504b08f7f09e355915ae19a1b25904 not found: ID does not exist" Jul 09 22:38:59.334753 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.334718 2568 scope.go:117] "RemoveContainer" containerID="bb0492fa481b3463e1aa9b83dc8c247971aa91a6df02222934c6c2c2e81201ab" Jul 09 22:38:59.334973 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:38:59.334957 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0492fa481b3463e1aa9b83dc8c247971aa91a6df02222934c6c2c2e81201ab\": container with ID starting with bb0492fa481b3463e1aa9b83dc8c247971aa91a6df02222934c6c2c2e81201ab not found: ID does not exist" containerID="bb0492fa481b3463e1aa9b83dc8c247971aa91a6df02222934c6c2c2e81201ab" Jul 09 22:38:59.335013 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.334977 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0492fa481b3463e1aa9b83dc8c247971aa91a6df02222934c6c2c2e81201ab"} err="failed to get container status \"bb0492fa481b3463e1aa9b83dc8c247971aa91a6df02222934c6c2c2e81201ab\": rpc error: code = NotFound desc = could not find container \"bb0492fa481b3463e1aa9b83dc8c247971aa91a6df02222934c6c2c2e81201ab\": container with ID starting with bb0492fa481b3463e1aa9b83dc8c247971aa91a6df02222934c6c2c2e81201ab not found: ID does not exist" Jul 09 22:38:59.335013 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.334992 2568 scope.go:117] "RemoveContainer" containerID="5d2945d02dd033c41ba8349c30e4274bc5954e452b1f78ec647f1bf6165cc788" Jul 09 22:38:59.335225 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:38:59.335207 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d2945d02dd033c41ba8349c30e4274bc5954e452b1f78ec647f1bf6165cc788\": container with ID starting with 5d2945d02dd033c41ba8349c30e4274bc5954e452b1f78ec647f1bf6165cc788 not found: ID does not exist" containerID="5d2945d02dd033c41ba8349c30e4274bc5954e452b1f78ec647f1bf6165cc788" Jul 09 22:38:59.335285 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.335228 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d2945d02dd033c41ba8349c30e4274bc5954e452b1f78ec647f1bf6165cc788"} err="failed to get container status \"5d2945d02dd033c41ba8349c30e4274bc5954e452b1f78ec647f1bf6165cc788\": rpc error: code = NotFound desc = could not find container \"5d2945d02dd033c41ba8349c30e4274bc5954e452b1f78ec647f1bf6165cc788\": container with ID starting with 5d2945d02dd033c41ba8349c30e4274bc5954e452b1f78ec647f1bf6165cc788 not found: ID does not exist" Jul 09 22:38:59.335285 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.335243 2568 scope.go:117] "RemoveContainer" containerID="c3bfbf17dd015c88cdef28788f8f2a1d7346e8c2bd2140c38faffcbff2e4a797" Jul 09 22:38:59.335437 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:38:59.335421 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3bfbf17dd015c88cdef28788f8f2a1d7346e8c2bd2140c38faffcbff2e4a797\": container with ID starting with c3bfbf17dd015c88cdef28788f8f2a1d7346e8c2bd2140c38faffcbff2e4a797 not found: ID does not exist" containerID="c3bfbf17dd015c88cdef28788f8f2a1d7346e8c2bd2140c38faffcbff2e4a797" Jul 09 22:38:59.335524 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.335439 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3bfbf17dd015c88cdef28788f8f2a1d7346e8c2bd2140c38faffcbff2e4a797"} err="failed to get container status \"c3bfbf17dd015c88cdef28788f8f2a1d7346e8c2bd2140c38faffcbff2e4a797\": rpc error: code = NotFound desc = could not find container \"c3bfbf17dd015c88cdef28788f8f2a1d7346e8c2bd2140c38faffcbff2e4a797\": container with ID starting with c3bfbf17dd015c88cdef28788f8f2a1d7346e8c2bd2140c38faffcbff2e4a797 not found: ID does not exist" Jul 09 22:38:59.335524 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.335453 2568 scope.go:117] "RemoveContainer" containerID="52cb2eff9019a0561c14b7b4ae99674801ac98e3d45ca2910040f7dbdaf6dccc" Jul 09 22:38:59.335669 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.335650 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52cb2eff9019a0561c14b7b4ae99674801ac98e3d45ca2910040f7dbdaf6dccc"} err="failed to get container status \"52cb2eff9019a0561c14b7b4ae99674801ac98e3d45ca2910040f7dbdaf6dccc\": rpc error: code = NotFound desc = could not find container \"52cb2eff9019a0561c14b7b4ae99674801ac98e3d45ca2910040f7dbdaf6dccc\": container with ID starting with 52cb2eff9019a0561c14b7b4ae99674801ac98e3d45ca2910040f7dbdaf6dccc not found: ID does not exist" Jul 09 22:38:59.335724 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.335669 2568 scope.go:117] "RemoveContainer" containerID="d5ea136bfeb1f55460396079096f00751cbbffcc334b9af2b4f2104424672478" Jul 09 22:38:59.335927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.335908 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5ea136bfeb1f55460396079096f00751cbbffcc334b9af2b4f2104424672478"} err="failed to get container status \"d5ea136bfeb1f55460396079096f00751cbbffcc334b9af2b4f2104424672478\": rpc error: code = NotFound desc = could not find container \"d5ea136bfeb1f55460396079096f00751cbbffcc334b9af2b4f2104424672478\": container with ID starting with d5ea136bfeb1f55460396079096f00751cbbffcc334b9af2b4f2104424672478 not found: ID does not exist" Jul 09 22:38:59.335997 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.335929 2568 scope.go:117] "RemoveContainer" containerID="55ee17369c5992e78344a4929395ef7d170493859125e16013502d7eac52e852" Jul 09 22:38:59.336155 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.336139 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55ee17369c5992e78344a4929395ef7d170493859125e16013502d7eac52e852"} err="failed to get container status \"55ee17369c5992e78344a4929395ef7d170493859125e16013502d7eac52e852\": rpc error: code = NotFound desc = could not find container \"55ee17369c5992e78344a4929395ef7d170493859125e16013502d7eac52e852\": container with ID starting with 55ee17369c5992e78344a4929395ef7d170493859125e16013502d7eac52e852 not found: ID does not exist" Jul 09 22:38:59.336202 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.336157 2568 scope.go:117] "RemoveContainer" containerID="71a6ecb1a1579358a9f97d9b4efaa10d4a504b08f7f09e355915ae19a1b25904" Jul 09 22:38:59.336347 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.336328 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a6ecb1a1579358a9f97d9b4efaa10d4a504b08f7f09e355915ae19a1b25904"} err="failed to get container status \"71a6ecb1a1579358a9f97d9b4efaa10d4a504b08f7f09e355915ae19a1b25904\": rpc error: code = NotFound desc = could not find container \"71a6ecb1a1579358a9f97d9b4efaa10d4a504b08f7f09e355915ae19a1b25904\": container with ID starting with 71a6ecb1a1579358a9f97d9b4efaa10d4a504b08f7f09e355915ae19a1b25904 not found: ID does not exist" Jul 09 22:38:59.336420 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.336348 2568 scope.go:117] "RemoveContainer" containerID="bb0492fa481b3463e1aa9b83dc8c247971aa91a6df02222934c6c2c2e81201ab" Jul 09 22:38:59.336594 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.336575 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0492fa481b3463e1aa9b83dc8c247971aa91a6df02222934c6c2c2e81201ab"} err="failed to get container status \"bb0492fa481b3463e1aa9b83dc8c247971aa91a6df02222934c6c2c2e81201ab\": rpc error: code = NotFound desc = could not find container \"bb0492fa481b3463e1aa9b83dc8c247971aa91a6df02222934c6c2c2e81201ab\": container with ID starting with bb0492fa481b3463e1aa9b83dc8c247971aa91a6df02222934c6c2c2e81201ab not found: ID does not exist" Jul 09 22:38:59.336652 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.336594 2568 scope.go:117] "RemoveContainer" containerID="5d2945d02dd033c41ba8349c30e4274bc5954e452b1f78ec647f1bf6165cc788" Jul 09 22:38:59.336800 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.336785 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d2945d02dd033c41ba8349c30e4274bc5954e452b1f78ec647f1bf6165cc788"} err="failed to get container status \"5d2945d02dd033c41ba8349c30e4274bc5954e452b1f78ec647f1bf6165cc788\": rpc error: code = NotFound desc = could not find container \"5d2945d02dd033c41ba8349c30e4274bc5954e452b1f78ec647f1bf6165cc788\": container with ID starting with 5d2945d02dd033c41ba8349c30e4274bc5954e452b1f78ec647f1bf6165cc788 not found: ID does not exist" Jul 09 22:38:59.336839 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.336799 2568 scope.go:117] "RemoveContainer" containerID="c3bfbf17dd015c88cdef28788f8f2a1d7346e8c2bd2140c38faffcbff2e4a797" Jul 09 22:38:59.337008 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.336991 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3bfbf17dd015c88cdef28788f8f2a1d7346e8c2bd2140c38faffcbff2e4a797"} err="failed to get container status \"c3bfbf17dd015c88cdef28788f8f2a1d7346e8c2bd2140c38faffcbff2e4a797\": rpc error: code = NotFound desc = could not find container \"c3bfbf17dd015c88cdef28788f8f2a1d7346e8c2bd2140c38faffcbff2e4a797\": container with ID starting with c3bfbf17dd015c88cdef28788f8f2a1d7346e8c2bd2140c38faffcbff2e4a797 not found: ID does not exist" Jul 09 22:38:59.413317 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.413281 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-main-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:38:59.413317 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.413312 2568 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-config-out\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:38:59.413317 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.413322 2568 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-web-config\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:38:59.413567 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.413332 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:38:59.413567 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.413342 2568 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:38:59.413567 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.413350 2568 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-cluster-tls-config\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:38:59.413567 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.413360 2568 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-tls-assets\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:38:59.413567 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.413369 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:38:59.413567 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.413378 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:38:59.413567 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.413386 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcpd2\" (UniqueName: \"kubernetes.io/projected/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-kube-api-access-bcpd2\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:38:59.413567 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.413394 2568 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ddab4fd6-05c7-41a3-b581-3ff08941fdaa-config-volume\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:38:59.594009 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.593972 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 22:38:59.604019 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.603978 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 22:38:59.630804 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.630772 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 22:38:59.631054 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631040 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab92fa47-2296-41b9-9636-b98ffd793b8a" containerName="registry" Jul 09 22:38:59.631099 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631057 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab92fa47-2296-41b9-9636-b98ffd793b8a" containerName="registry" Jul 09 22:38:59.631099 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631069 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="alertmanager" Jul 09 22:38:59.631099 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631076 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="alertmanager" Jul 09 22:38:59.631099 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631083 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="prom-label-proxy" Jul 09 22:38:59.631099 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631088 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="prom-label-proxy" Jul 09 22:38:59.631099 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631097 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="config-reloader" Jul 09 22:38:59.631099 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631102 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="config-reloader" Jul 09 22:38:59.631298 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631111 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="init-config-reloader" Jul 09 22:38:59.631298 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631117 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="init-config-reloader" Jul 09 22:38:59.631298 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631122 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="kube-rbac-proxy-web" Jul 09 22:38:59.631298 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631127 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="kube-rbac-proxy-web" Jul 09 22:38:59.631298 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631146 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="kube-rbac-proxy" Jul 09 22:38:59.631298 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631154 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="kube-rbac-proxy" Jul 09 22:38:59.631298 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631160 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="kube-rbac-proxy-metric" Jul 09 22:38:59.631298 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631165 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="kube-rbac-proxy-metric" Jul 09 22:38:59.631298 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631211 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="alertmanager" Jul 09 22:38:59.631298 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631218 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="config-reloader" Jul 09 22:38:59.631298 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631228 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="kube-rbac-proxy" Jul 09 22:38:59.631298 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631237 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="kube-rbac-proxy-metric" Jul 09 22:38:59.631298 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631242 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab92fa47-2296-41b9-9636-b98ffd793b8a" containerName="registry" Jul 09 22:38:59.631298 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631247 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="kube-rbac-proxy-web" Jul 09 22:38:59.631298 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.631252 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" containerName="prom-label-proxy" Jul 09 22:38:59.636001 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.635975 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.638892 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.638861 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 09 22:38:59.638892 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.638864 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 09 22:38:59.639133 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.638866 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mhj2k\"" Jul 09 22:38:59.639133 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.639031 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 09 22:38:59.639242 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.639198 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 09 22:38:59.639242 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.639198 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 09 22:38:59.639242 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.639225 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 09 22:38:59.639242 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.639233 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 09 22:38:59.639423 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.639259 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 09 22:38:59.646002 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.645978 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 09 22:38:59.652719 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.652688 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 22:38:59.715274 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.715236 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2l4v\" (UniqueName: \"kubernetes.io/projected/273c3606-e9c2-49da-b58d-a26c08314372-kube-api-access-w2l4v\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.715457 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.715290 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.715457 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.715354 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/273c3606-e9c2-49da-b58d-a26c08314372-config-out\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.715457 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.715389 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.715457 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.715420 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/273c3606-e9c2-49da-b58d-a26c08314372-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.715457 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.715437 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-web-config\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.715659 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.715462 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-config-volume\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.715659 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.715481 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/273c3606-e9c2-49da-b58d-a26c08314372-tls-assets\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.715659 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.715524 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.715659 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.715546 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.715659 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.715562 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/273c3606-e9c2-49da-b58d-a26c08314372-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.715659 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.715594 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.715659 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.715637 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/273c3606-e9c2-49da-b58d-a26c08314372-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.816709 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.816674 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/273c3606-e9c2-49da-b58d-a26c08314372-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.816709 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.816714 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2l4v\" (UniqueName: \"kubernetes.io/projected/273c3606-e9c2-49da-b58d-a26c08314372-kube-api-access-w2l4v\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.816932 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.816743 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.816932 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.816762 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/273c3606-e9c2-49da-b58d-a26c08314372-config-out\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.816932 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.816781 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.816932 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.816804 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/273c3606-e9c2-49da-b58d-a26c08314372-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.816932 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.816819 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-web-config\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.816932 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.816833 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-config-volume\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.816932 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.816857 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/273c3606-e9c2-49da-b58d-a26c08314372-tls-assets\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.817257 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.817125 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/273c3606-e9c2-49da-b58d-a26c08314372-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.817311 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.817262 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.817311 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.817302 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.817414 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.817330 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/273c3606-e9c2-49da-b58d-a26c08314372-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.817414 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.817373 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.817689 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.817669 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/273c3606-e9c2-49da-b58d-a26c08314372-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.819892 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.819846 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.820000 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.819906 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/273c3606-e9c2-49da-b58d-a26c08314372-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.820000 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.819969 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-config-volume\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.820396 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.820374 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/273c3606-e9c2-49da-b58d-a26c08314372-config-out\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.820482 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.820439 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/273c3606-e9c2-49da-b58d-a26c08314372-tls-assets\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.820482 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.820440 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.820575 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.820561 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-web-config\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.820651 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.820636 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.821081 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.821064 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.821464 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.821446 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/273c3606-e9c2-49da-b58d-a26c08314372-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.826036 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.826009 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2l4v\" (UniqueName: \"kubernetes.io/projected/273c3606-e9c2-49da-b58d-a26c08314372-kube-api-access-w2l4v\") pod \"alertmanager-main-0\" (UID: \"273c3606-e9c2-49da-b58d-a26c08314372\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:38:59.946106 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:38:59.946023 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 22:39:00.048341 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:00.048310 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddab4fd6-05c7-41a3-b581-3ff08941fdaa" path="/var/lib/kubelet/pods/ddab4fd6-05c7-41a3-b581-3ff08941fdaa/volumes" Jul 09 22:39:00.082297 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:00.082257 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 22:39:00.085294 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:39:00.085261 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273c3606_e9c2_49da_b58d_a26c08314372.slice/crio-80b1ecd1015cefdba18491afec72ada43b54f47f5c488ba0fe187d1c03fc12fe WatchSource:0}: Error finding container 80b1ecd1015cefdba18491afec72ada43b54f47f5c488ba0fe187d1c03fc12fe: Status 404 returned error can't find the container with id 80b1ecd1015cefdba18491afec72ada43b54f47f5c488ba0fe187d1c03fc12fe Jul 09 22:39:00.275751 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:00.275652 2568 generic.go:356] "Generic (PLEG): container finished" podID="273c3606-e9c2-49da-b58d-a26c08314372" containerID="c36205947e82a0806b5e6ef8130a1c4092921f1336e663f213f869e5d20ce8f9" exitCode=0 Jul 09 22:39:00.276154 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:00.275749 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"273c3606-e9c2-49da-b58d-a26c08314372","Type":"ContainerDied","Data":"c36205947e82a0806b5e6ef8130a1c4092921f1336e663f213f869e5d20ce8f9"} Jul 09 22:39:00.276154 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:00.275829 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"273c3606-e9c2-49da-b58d-a26c08314372","Type":"ContainerStarted","Data":"80b1ecd1015cefdba18491afec72ada43b54f47f5c488ba0fe187d1c03fc12fe"} Jul 09 22:39:01.283873 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:01.283837 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"273c3606-e9c2-49da-b58d-a26c08314372","Type":"ContainerStarted","Data":"1ecbfda459742ebbfc2e4d4128915f6e8734e791e443d179c421b485a120f813"} Jul 09 22:39:01.284293 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:01.283936 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"273c3606-e9c2-49da-b58d-a26c08314372","Type":"ContainerStarted","Data":"3833dc1fceee5908193df0e0ff7cc545153695e3b4a5055939e59cc87a13dc0d"} Jul 09 22:39:01.284293 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:01.283963 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"273c3606-e9c2-49da-b58d-a26c08314372","Type":"ContainerStarted","Data":"50c82054d0cbf61ce5aece05488a3cf23e9c0b47e5853073bbfb9d37c8113fb3"} Jul 09 22:39:01.284293 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:01.283988 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"273c3606-e9c2-49da-b58d-a26c08314372","Type":"ContainerStarted","Data":"94ca047aeba2e1578236a27427e0c711cb317085af25cd182eed6e7b1dfa64f1"} Jul 09 22:39:01.284293 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:01.284011 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"273c3606-e9c2-49da-b58d-a26c08314372","Type":"ContainerStarted","Data":"4bd3a871bcddd3d5074f96c05c3e7ce261062a071f5db231c82cdb4de697f6d1"} Jul 09 22:39:01.284293 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:01.284035 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"273c3606-e9c2-49da-b58d-a26c08314372","Type":"ContainerStarted","Data":"c9846661b0c684ed24cbbe498971ff37e43ff3419ec51a8614798eb11672d8f2"} Jul 09 22:39:01.313086 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:01.313025 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.29738099 podStartE2EDuration="2.313008865s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.627875ms" isStatefulPod=false podCreationTimestamp="2026-07-09 22:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:39:01.311435846 +0000 UTC m=+365.852514640" watchObservedRunningTime="2026-07-09 22:39:01.313008865 +0000 UTC m=+365.854087658" Jul 09 22:39:02.035400 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.035366 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7597d88677-qmmsm"] Jul 09 22:39:02.038740 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.038722 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.041814 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.041781 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-pptbd\"" Jul 09 22:39:02.041962 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.041785 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 09 22:39:02.041962 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.041895 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 09 22:39:02.042132 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.042117 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 09 22:39:02.042228 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.042212 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 09 22:39:02.042427 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.042415 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 09 22:39:02.048094 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.048068 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 09 22:39:02.054365 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.054339 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7597d88677-qmmsm"] Jul 09 22:39:02.134909 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.134862 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2e5ee69-00e6-4411-9b81-fece72803fb2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.134909 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.134915 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2e5ee69-00e6-4411-9b81-fece72803fb2-serving-certs-ca-bundle\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.135150 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.134954 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a2e5ee69-00e6-4411-9b81-fece72803fb2-telemeter-client-tls\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.135150 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.134981 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2e5ee69-00e6-4411-9b81-fece72803fb2-metrics-client-ca\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.135150 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.134998 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a2e5ee69-00e6-4411-9b81-fece72803fb2-secret-telemeter-client\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.135150 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.135055 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a2e5ee69-00e6-4411-9b81-fece72803fb2-federate-client-tls\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.135150 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.135110 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2e5ee69-00e6-4411-9b81-fece72803fb2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.135150 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.135133 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdxtm\" (UniqueName: \"kubernetes.io/projected/a2e5ee69-00e6-4411-9b81-fece72803fb2-kube-api-access-vdxtm\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.236074 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.236039 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2e5ee69-00e6-4411-9b81-fece72803fb2-metrics-client-ca\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.236074 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.236074 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a2e5ee69-00e6-4411-9b81-fece72803fb2-secret-telemeter-client\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.236287 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.236104 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a2e5ee69-00e6-4411-9b81-fece72803fb2-federate-client-tls\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.236287 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.236143 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2e5ee69-00e6-4411-9b81-fece72803fb2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.236287 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.236158 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdxtm\" (UniqueName: \"kubernetes.io/projected/a2e5ee69-00e6-4411-9b81-fece72803fb2-kube-api-access-vdxtm\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.236446 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.236312 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2e5ee69-00e6-4411-9b81-fece72803fb2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.236446 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.236391 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2e5ee69-00e6-4411-9b81-fece72803fb2-serving-certs-ca-bundle\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.236778 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.236749 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a2e5ee69-00e6-4411-9b81-fece72803fb2-telemeter-client-tls\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.236958 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.236930 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2e5ee69-00e6-4411-9b81-fece72803fb2-metrics-client-ca\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.237090 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.237072 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2e5ee69-00e6-4411-9b81-fece72803fb2-serving-certs-ca-bundle\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.237180 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.237090 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2e5ee69-00e6-4411-9b81-fece72803fb2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.238853 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.238827 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a2e5ee69-00e6-4411-9b81-fece72803fb2-federate-client-tls\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.238853 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.238849 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a2e5ee69-00e6-4411-9b81-fece72803fb2-telemeter-client-tls\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.239116 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.239099 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2e5ee69-00e6-4411-9b81-fece72803fb2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.239303 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.239281 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a2e5ee69-00e6-4411-9b81-fece72803fb2-secret-telemeter-client\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.244533 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.244509 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdxtm\" (UniqueName: \"kubernetes.io/projected/a2e5ee69-00e6-4411-9b81-fece72803fb2-kube-api-access-vdxtm\") pod \"telemeter-client-7597d88677-qmmsm\" (UID: \"a2e5ee69-00e6-4411-9b81-fece72803fb2\") " pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.350452 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.350406 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" Jul 09 22:39:02.491995 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.491960 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7597d88677-qmmsm"] Jul 09 22:39:02.495010 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:39:02.494983 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e5ee69_00e6_4411_9b81_fece72803fb2.slice/crio-446c5500c0fba8718778528673f38e881d62d11f7489d68731dadab07e583581 WatchSource:0}: Error finding container 446c5500c0fba8718778528673f38e881d62d11f7489d68731dadab07e583581: Status 404 returned error can't find the container with id 446c5500c0fba8718778528673f38e881d62d11f7489d68731dadab07e583581 Jul 09 22:39:02.496995 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:02.496978 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 22:39:03.292855 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:03.292820 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" event={"ID":"a2e5ee69-00e6-4411-9b81-fece72803fb2","Type":"ContainerStarted","Data":"446c5500c0fba8718778528673f38e881d62d11f7489d68731dadab07e583581"} Jul 09 22:39:05.303719 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:05.303676 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" event={"ID":"a2e5ee69-00e6-4411-9b81-fece72803fb2","Type":"ContainerStarted","Data":"93a871462a94dfab591e0f1f8e968da35aaa08485448b9aee1ffe82ecb56d53a"} Jul 09 22:39:05.304129 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:05.303752 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" event={"ID":"a2e5ee69-00e6-4411-9b81-fece72803fb2","Type":"ContainerStarted","Data":"ea73f75c1946846d615e6c6b63b095253a532d71d6b5d66492f2e9a5c27f6423"} Jul 09 22:39:05.304129 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:05.303780 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" event={"ID":"a2e5ee69-00e6-4411-9b81-fece72803fb2","Type":"ContainerStarted","Data":"ca84b2f2e55af838b5fbb6a8826c8e231e4f79eb3728a8788974061fb65323fc"} Jul 09 22:39:05.329267 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:05.329213 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7597d88677-qmmsm" podStartSLOduration=1.383757903 podStartE2EDuration="3.329196741s" totalImagesPullingTime="1.945438838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:39:05.328250184 +0000 UTC m=+369.869328978" watchObservedRunningTime="2026-07-09 22:39:05.329196741 +0000 UTC m=+369.870275534" Jul 09 22:39:06.440546 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:06.440512 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:39:09.903516 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:09.903449 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs\") pod \"network-metrics-daemon-5wch9\" (UID: \"941cb948-d902-4c71-abf2-52ec25863dd8\") " pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:39:09.905718 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:09.905697 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941cb948-d902-4c71-abf2-52ec25863dd8-metrics-certs\") pod \"network-metrics-daemon-5wch9\" (UID: \"941cb948-d902-4c71-abf2-52ec25863dd8\") " pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:39:10.045942 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:10.045915 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jjswb\"" Jul 09 22:39:10.053379 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:10.053350 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wch9" Jul 09 22:39:10.181069 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:10.181042 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5wch9"] Jul 09 22:39:10.183146 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:39:10.183124 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941cb948_d902_4c71_abf2_52ec25863dd8.slice/crio-201377474ce360aa3967a917f23b6d9e81db576de45c96249c5e0ef066db496b WatchSource:0}: Error finding container 201377474ce360aa3967a917f23b6d9e81db576de45c96249c5e0ef066db496b: Status 404 returned error can't find the container with id 201377474ce360aa3967a917f23b6d9e81db576de45c96249c5e0ef066db496b Jul 09 22:39:10.323960 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:10.323924 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5wch9" event={"ID":"941cb948-d902-4c71-abf2-52ec25863dd8","Type":"ContainerStarted","Data":"201377474ce360aa3967a917f23b6d9e81db576de45c96249c5e0ef066db496b"} Jul 09 22:39:11.329470 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:11.329441 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5wch9" event={"ID":"941cb948-d902-4c71-abf2-52ec25863dd8","Type":"ContainerStarted","Data":"5a5717f605571d10757e271144d2d06b9946272078c4f2e8b460c43fefd36b9d"} Jul 09 22:39:12.334449 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:12.334414 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5wch9" event={"ID":"941cb948-d902-4c71-abf2-52ec25863dd8","Type":"ContainerStarted","Data":"802a9e139a7fcf730a7338d55bd82ff78f3dc04f681d92bd74aa955cbdd52d5c"} Jul 09 22:39:12.354462 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:12.354405 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5wch9" podStartSLOduration=375.378513171 podStartE2EDuration="6m16.35438244s" totalImagesPullingTime="975.869269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:39:12.352218742 +0000 UTC m=+376.893297535" watchObservedRunningTime="2026-07-09 22:39:12.35438244 +0000 UTC m=+376.895461234" Jul 09 22:39:12.452044 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:12.452006 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:39:12.556926 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:12.556892 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:39:17.510940 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:17.510900 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:39:17.611416 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:17.611384 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:39:17.713101 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:17.713071 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:39:59.477989 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:39:59.477950 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:40:12.178999 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:12.178962 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:40:12.237833 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:12.237800 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:40:16.199178 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:16.199142 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:40:16.268886 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:16.268849 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:40:16.370948 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:16.370916 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:40:16.939166 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:16.939133 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:40:16.995887 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:16.995831 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:40:19.488707 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:19.488671 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:40:24.021468 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:24.021432 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:40:24.122650 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:24.122616 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:40:26.716123 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:26.716088 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:40:26.771975 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:26.771941 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:40:26.830947 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:26.830919 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:40:28.366204 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:28.366170 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:40:28.466834 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:28.466803 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:40:28.711285 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:28.711172 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:40:53.170387 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:53.170336 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:40:53.266207 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:53.266159 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:40:53.899582 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:53.899548 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl"] Jul 09 22:40:53.902649 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:53.902626 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl" Jul 09 22:40:53.905537 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:53.905513 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 22:40:53.906964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:53.906944 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 22:40:53.907094 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:53.907031 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-llmcv\"" Jul 09 22:40:53.912189 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:53.912165 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl"] Jul 09 22:40:53.994183 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:53.994143 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl\" (UID: \"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl" Jul 09 22:40:53.994380 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:53.994198 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl\" (UID: \"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl" Jul 09 22:40:53.994380 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:53.994283 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqnn9\" (UniqueName: \"kubernetes.io/projected/a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d-kube-api-access-cqnn9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl\" (UID: \"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl" Jul 09 22:40:54.094907 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:54.094865 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl\" (UID: \"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl" Jul 09 22:40:54.095137 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:54.094914 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl\" (UID: \"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl" Jul 09 22:40:54.095137 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:54.094992 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqnn9\" (UniqueName: \"kubernetes.io/projected/a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d-kube-api-access-cqnn9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl\" (UID: \"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl" Jul 09 22:40:54.095296 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:54.095274 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl\" (UID: \"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl" Jul 09 22:40:54.095385 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:54.095362 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl\" (UID: \"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl" Jul 09 22:40:54.103820 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:54.103789 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqnn9\" (UniqueName: \"kubernetes.io/projected/a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d-kube-api-access-cqnn9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl\" (UID: \"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl" Jul 09 22:40:54.212809 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:54.212715 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl" Jul 09 22:40:54.342672 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:54.342640 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl"] Jul 09 22:40:54.345837 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:40:54.345810 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f5bf14_4ac3_45d4_bf0c_4d4823fd609d.slice/crio-f30e976f3b2c5f27b99544030dba103aad5291cefb3c08f5fb44dfd91a1fe298 WatchSource:0}: Error finding container f30e976f3b2c5f27b99544030dba103aad5291cefb3c08f5fb44dfd91a1fe298: Status 404 returned error can't find the container with id f30e976f3b2c5f27b99544030dba103aad5291cefb3c08f5fb44dfd91a1fe298 Jul 09 22:40:54.702895 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:40:54.702855 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl" event={"ID":"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d","Type":"ContainerStarted","Data":"f30e976f3b2c5f27b99544030dba103aad5291cefb3c08f5fb44dfd91a1fe298"} Jul 09 22:41:01.733666 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:01.733630 2568 generic.go:356] "Generic (PLEG): container finished" podID="a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d" containerID="3b17e6b58c87e60b60a446f90cd0f2032e647908664984fde58a1eb1dc2c2d0d" exitCode=0 Jul 09 22:41:01.734067 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:01.733721 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl" event={"ID":"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d","Type":"ContainerDied","Data":"3b17e6b58c87e60b60a446f90cd0f2032e647908664984fde58a1eb1dc2c2d0d"} Jul 09 22:41:04.746252 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:04.746216 2568 generic.go:356] "Generic (PLEG): container finished" podID="a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d" containerID="f758631820b8b27c13397b7b7f7624674d26733e73508891d036ac71ef28b87b" exitCode=0 Jul 09 22:41:04.746760 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:04.746277 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl" event={"ID":"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d","Type":"ContainerDied","Data":"f758631820b8b27c13397b7b7f7624674d26733e73508891d036ac71ef28b87b"} Jul 09 22:41:13.782855 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:13.782818 2568 generic.go:356] "Generic (PLEG): container finished" podID="a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d" containerID="d4d8db5c576f58c436fa8df2b14f2efb83f41e84de807a27df105c7f5fb23c28" exitCode=0 Jul 09 22:41:13.783261 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:13.782913 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl" event={"ID":"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d","Type":"ContainerDied","Data":"d4d8db5c576f58c436fa8df2b14f2efb83f41e84de807a27df105c7f5fb23c28"} Jul 09 22:41:14.910389 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:14.910359 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl" Jul 09 22:41:14.975708 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:14.975666 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d-bundle\") pod \"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d\" (UID: \"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d\") " Jul 09 22:41:14.975708 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:14.975711 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d-util\") pod \"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d\" (UID: \"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d\") " Jul 09 22:41:14.975957 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:14.975743 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqnn9\" (UniqueName: \"kubernetes.io/projected/a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d-kube-api-access-cqnn9\") pod \"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d\" (UID: \"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d\") " Jul 09 22:41:14.976390 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:14.976357 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d-bundle" (OuterVolumeSpecName: "bundle") pod "a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d" (UID: "a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:41:14.977933 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:14.977909 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d-kube-api-access-cqnn9" (OuterVolumeSpecName: "kube-api-access-cqnn9") pod "a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d" (UID: "a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d"). InnerVolumeSpecName "kube-api-access-cqnn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:41:14.979702 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:14.979679 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d-util" (OuterVolumeSpecName: "util") pod "a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d" (UID: "a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:41:15.076949 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:15.076895 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqnn9\" (UniqueName: \"kubernetes.io/projected/a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d-kube-api-access-cqnn9\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:41:15.076949 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:15.076939 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d-bundle\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:41:15.076949 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:15.076950 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d-util\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:41:15.792220 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:15.792181 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl" event={"ID":"a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d","Type":"ContainerDied","Data":"f30e976f3b2c5f27b99544030dba103aad5291cefb3c08f5fb44dfd91a1fe298"} Jul 09 22:41:15.792220 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:15.792204 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8pjl" Jul 09 22:41:15.792427 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:15.792235 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f30e976f3b2c5f27b99544030dba103aad5291cefb3c08f5fb44dfd91a1fe298" Jul 09 22:41:19.939119 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:19.939074 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:41:20.796197 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:20.796157 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:41:21.939459 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:21.939421 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4qnx"] Jul 09 22:41:21.939945 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:21.939813 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d" containerName="pull" Jul 09 22:41:21.939945 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:21.939829 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d" containerName="pull" Jul 09 22:41:21.939945 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:21.939849 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d" containerName="extract" Jul 09 22:41:21.939945 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:21.939856 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d" containerName="extract" Jul 09 22:41:21.939945 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:21.939876 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d" containerName="util" Jul 09 22:41:21.939945 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:21.939884 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d" containerName="util" Jul 09 22:41:21.939945 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:21.939945 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1f5bf14-4ac3-45d4-bf0c-4d4823fd609d" containerName="extract" Jul 09 22:41:21.942793 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:21.942770 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4qnx" Jul 09 22:41:21.945722 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:21.945701 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-z457b\"" Jul 09 22:41:21.945841 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:21.945732 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 09 22:41:21.945841 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:21.945732 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 09 22:41:21.959413 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:21.959387 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4qnx"] Jul 09 22:41:22.031961 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:22.031921 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxpjp\" (UniqueName: \"kubernetes.io/projected/471e91e2-4fbc-4ee7-a319-5922c17510de-kube-api-access-nxpjp\") pod \"cert-manager-operator-controller-manager-55c444bf86-m4qnx\" (UID: \"471e91e2-4fbc-4ee7-a319-5922c17510de\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4qnx" Jul 09 22:41:22.031961 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:22.031961 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/471e91e2-4fbc-4ee7-a319-5922c17510de-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-m4qnx\" (UID: \"471e91e2-4fbc-4ee7-a319-5922c17510de\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4qnx" Jul 09 22:41:22.133309 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:22.133266 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpjp\" (UniqueName: \"kubernetes.io/projected/471e91e2-4fbc-4ee7-a319-5922c17510de-kube-api-access-nxpjp\") pod \"cert-manager-operator-controller-manager-55c444bf86-m4qnx\" (UID: \"471e91e2-4fbc-4ee7-a319-5922c17510de\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4qnx" Jul 09 22:41:22.133309 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:22.133309 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/471e91e2-4fbc-4ee7-a319-5922c17510de-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-m4qnx\" (UID: \"471e91e2-4fbc-4ee7-a319-5922c17510de\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4qnx" Jul 09 22:41:22.133783 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:22.133754 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/471e91e2-4fbc-4ee7-a319-5922c17510de-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-m4qnx\" (UID: \"471e91e2-4fbc-4ee7-a319-5922c17510de\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4qnx" Jul 09 22:41:22.145398 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:22.145363 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxpjp\" (UniqueName: \"kubernetes.io/projected/471e91e2-4fbc-4ee7-a319-5922c17510de-kube-api-access-nxpjp\") pod \"cert-manager-operator-controller-manager-55c444bf86-m4qnx\" (UID: \"471e91e2-4fbc-4ee7-a319-5922c17510de\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4qnx" Jul 09 22:41:22.256747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:22.256650 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4qnx" Jul 09 22:41:22.389862 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:22.389831 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4qnx"] Jul 09 22:41:22.393844 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:41:22.393819 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471e91e2_4fbc_4ee7_a319_5922c17510de.slice/crio-a4bae56208b1b0f2692f827bbe6ef7701ebf1958ffbc5880ee78f50e24860c14 WatchSource:0}: Error finding container a4bae56208b1b0f2692f827bbe6ef7701ebf1958ffbc5880ee78f50e24860c14: Status 404 returned error can't find the container with id a4bae56208b1b0f2692f827bbe6ef7701ebf1958ffbc5880ee78f50e24860c14 Jul 09 22:41:22.818444 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:22.818409 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4qnx" event={"ID":"471e91e2-4fbc-4ee7-a319-5922c17510de","Type":"ContainerStarted","Data":"a4bae56208b1b0f2692f827bbe6ef7701ebf1958ffbc5880ee78f50e24860c14"} Jul 09 22:41:23.573306 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:23.573266 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:41:23.682249 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:23.682210 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:41:24.828329 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:24.828293 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4qnx" event={"ID":"471e91e2-4fbc-4ee7-a319-5922c17510de","Type":"ContainerStarted","Data":"72ef586eaae0f4412feec50d1e30365b38ee074a7db9ae5ef66fecd46aba21e6"} Jul 09 22:41:24.852725 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:24.852671 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m4qnx" podStartSLOduration=2.045321811 podStartE2EDuration="3.852653277s" totalImagesPullingTime="1.807331466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:41:24.851257652 +0000 UTC m=+509.392336445" watchObservedRunningTime="2026-07-09 22:41:24.852653277 +0000 UTC m=+509.393732067" Jul 09 22:41:25.159952 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:25.159861 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:41:25.270751 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:25.270710 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:41:25.395607 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:25.395567 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:41:26.186757 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.186725 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb"] Jul 09 22:41:26.190707 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.190691 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb" Jul 09 22:41:26.194232 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.194214 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 22:41:26.195471 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.195454 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 22:41:26.195572 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.195474 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-llmcv\"" Jul 09 22:41:26.204237 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.204211 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb"] Jul 09 22:41:26.269672 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.269640 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e728996-2f0e-4af9-a6ff-c01f09b0cc17-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb\" (UID: \"0e728996-2f0e-4af9-a6ff-c01f09b0cc17\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb" Jul 09 22:41:26.269844 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.269680 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltmbg\" (UniqueName: \"kubernetes.io/projected/0e728996-2f0e-4af9-a6ff-c01f09b0cc17-kube-api-access-ltmbg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb\" (UID: \"0e728996-2f0e-4af9-a6ff-c01f09b0cc17\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb" Jul 09 22:41:26.269844 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.269711 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e728996-2f0e-4af9-a6ff-c01f09b0cc17-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb\" (UID: \"0e728996-2f0e-4af9-a6ff-c01f09b0cc17\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb" Jul 09 22:41:26.370276 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.370244 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e728996-2f0e-4af9-a6ff-c01f09b0cc17-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb\" (UID: \"0e728996-2f0e-4af9-a6ff-c01f09b0cc17\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb" Jul 09 22:41:26.370456 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.370312 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e728996-2f0e-4af9-a6ff-c01f09b0cc17-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb\" (UID: \"0e728996-2f0e-4af9-a6ff-c01f09b0cc17\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb" Jul 09 22:41:26.370456 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.370404 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltmbg\" (UniqueName: \"kubernetes.io/projected/0e728996-2f0e-4af9-a6ff-c01f09b0cc17-kube-api-access-ltmbg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb\" (UID: \"0e728996-2f0e-4af9-a6ff-c01f09b0cc17\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb" Jul 09 22:41:26.370718 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.370699 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e728996-2f0e-4af9-a6ff-c01f09b0cc17-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb\" (UID: \"0e728996-2f0e-4af9-a6ff-c01f09b0cc17\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb" Jul 09 22:41:26.370760 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.370725 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e728996-2f0e-4af9-a6ff-c01f09b0cc17-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb\" (UID: \"0e728996-2f0e-4af9-a6ff-c01f09b0cc17\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb" Jul 09 22:41:26.379293 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.379260 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltmbg\" (UniqueName: \"kubernetes.io/projected/0e728996-2f0e-4af9-a6ff-c01f09b0cc17-kube-api-access-ltmbg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb\" (UID: \"0e728996-2f0e-4af9-a6ff-c01f09b0cc17\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb" Jul 09 22:41:26.499596 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.499512 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb" Jul 09 22:41:26.626813 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.626791 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb"] Jul 09 22:41:26.629520 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:41:26.629475 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e728996_2f0e_4af9_a6ff_c01f09b0cc17.slice/crio-8ea199da68dd640c295270c052d500ce8b6725bd8e4b559cbe46fd68e13ff38b WatchSource:0}: Error finding container 8ea199da68dd640c295270c052d500ce8b6725bd8e4b559cbe46fd68e13ff38b: Status 404 returned error can't find the container with id 8ea199da68dd640c295270c052d500ce8b6725bd8e4b559cbe46fd68e13ff38b Jul 09 22:41:26.837096 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.837060 2568 generic.go:356] "Generic (PLEG): container finished" podID="0e728996-2f0e-4af9-a6ff-c01f09b0cc17" containerID="d7e7995d53b5fd50a03bb2afcf664d2bb5b0f709470e94cd1e38a8e2cfd00e08" exitCode=0 Jul 09 22:41:26.837355 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.837143 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb" event={"ID":"0e728996-2f0e-4af9-a6ff-c01f09b0cc17","Type":"ContainerDied","Data":"d7e7995d53b5fd50a03bb2afcf664d2bb5b0f709470e94cd1e38a8e2cfd00e08"} Jul 09 22:41:26.837355 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:26.837226 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb" event={"ID":"0e728996-2f0e-4af9-a6ff-c01f09b0cc17","Type":"ContainerStarted","Data":"8ea199da68dd640c295270c052d500ce8b6725bd8e4b559cbe46fd68e13ff38b"} Jul 09 22:41:27.581981 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:27.581940 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:41:27.978849 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:27.978818 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:41:30.340037 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:30.340000 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:41:30.448458 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:30.448413 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:41:34.981302 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:34.981255 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:41:35.523538 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:35.523502 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-q6bqn"] Jul 09 22:41:35.576014 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:35.575979 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-q6bqn"] Jul 09 22:41:35.576184 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:35.576074 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-q6bqn" Jul 09 22:41:35.579835 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:35.579813 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hsp9l\"" Jul 09 22:41:35.580712 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:35.580691 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 09 22:41:35.581199 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:35.581186 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 09 22:41:35.748111 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:35.748074 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f085e495-885e-4410-a566-a677a90efe47-bound-sa-token\") pod \"cert-manager-55b4d954d5-q6bqn\" (UID: \"f085e495-885e-4410-a566-a677a90efe47\") " pod="cert-manager/cert-manager-55b4d954d5-q6bqn" Jul 09 22:41:35.748310 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:35.748145 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqf9d\" (UniqueName: \"kubernetes.io/projected/f085e495-885e-4410-a566-a677a90efe47-kube-api-access-mqf9d\") pod \"cert-manager-55b4d954d5-q6bqn\" (UID: \"f085e495-885e-4410-a566-a677a90efe47\") " pod="cert-manager/cert-manager-55b4d954d5-q6bqn" Jul 09 22:41:35.849055 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:35.849022 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqf9d\" (UniqueName: \"kubernetes.io/projected/f085e495-885e-4410-a566-a677a90efe47-kube-api-access-mqf9d\") pod \"cert-manager-55b4d954d5-q6bqn\" (UID: \"f085e495-885e-4410-a566-a677a90efe47\") " pod="cert-manager/cert-manager-55b4d954d5-q6bqn" Jul 09 22:41:35.849255 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:35.849072 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f085e495-885e-4410-a566-a677a90efe47-bound-sa-token\") pod \"cert-manager-55b4d954d5-q6bqn\" (UID: \"f085e495-885e-4410-a566-a677a90efe47\") " pod="cert-manager/cert-manager-55b4d954d5-q6bqn" Jul 09 22:41:35.857709 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:35.857679 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f085e495-885e-4410-a566-a677a90efe47-bound-sa-token\") pod \"cert-manager-55b4d954d5-q6bqn\" (UID: \"f085e495-885e-4410-a566-a677a90efe47\") " pod="cert-manager/cert-manager-55b4d954d5-q6bqn" Jul 09 22:41:35.857865 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:35.857844 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqf9d\" (UniqueName: \"kubernetes.io/projected/f085e495-885e-4410-a566-a677a90efe47-kube-api-access-mqf9d\") pod \"cert-manager-55b4d954d5-q6bqn\" (UID: \"f085e495-885e-4410-a566-a677a90efe47\") " pod="cert-manager/cert-manager-55b4d954d5-q6bqn" Jul 09 22:41:35.885867 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:35.885834 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-q6bqn" Jul 09 22:41:36.041714 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:36.041683 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-q6bqn"] Jul 09 22:41:36.044446 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:41:36.044416 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf085e495_885e_4410_a566_a677a90efe47.slice/crio-b5ec6ec458065c2878e500e4b1bd59087d39ebdd6099da7cc0bf67070858f169 WatchSource:0}: Error finding container b5ec6ec458065c2878e500e4b1bd59087d39ebdd6099da7cc0bf67070858f169: Status 404 returned error can't find the container with id b5ec6ec458065c2878e500e4b1bd59087d39ebdd6099da7cc0bf67070858f169 Jul 09 22:41:36.881453 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:36.881413 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-q6bqn" event={"ID":"f085e495-885e-4410-a566-a677a90efe47","Type":"ContainerStarted","Data":"b5ec6ec458065c2878e500e4b1bd59087d39ebdd6099da7cc0bf67070858f169"} Jul 09 22:41:39.896560 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:39.896525 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-q6bqn" event={"ID":"f085e495-885e-4410-a566-a677a90efe47","Type":"ContainerStarted","Data":"1ec4212678a7d38dd0ec867954e9aa4b8822f7bd01829cd0dc8bf131a2685b74"} Jul 09 22:41:43.916014 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:43.915981 2568 generic.go:356] "Generic (PLEG): container finished" podID="0e728996-2f0e-4af9-a6ff-c01f09b0cc17" containerID="42a5680e221d2c9d33c9bf9d2936c97ae16e717c3fc925486ed78f2e2813b52f" exitCode=0 Jul 09 22:41:43.916439 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:43.916021 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb" event={"ID":"0e728996-2f0e-4af9-a6ff-c01f09b0cc17","Type":"ContainerDied","Data":"42a5680e221d2c9d33c9bf9d2936c97ae16e717c3fc925486ed78f2e2813b52f"} Jul 09 22:41:43.937909 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:43.937845 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-q6bqn" podStartSLOduration=5.823810395 podStartE2EDuration="8.937823922s" totalImagesPullingTime="3.114013527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:41:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:41:39.91371768 +0000 UTC m=+524.454796500" watchObservedRunningTime="2026-07-09 22:41:43.937823922 +0000 UTC m=+528.478902716" Jul 09 22:41:44.922462 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:44.922421 2568 generic.go:356] "Generic (PLEG): container finished" podID="0e728996-2f0e-4af9-a6ff-c01f09b0cc17" containerID="14883d16a132c4785180136853cb37b624e26443bfd5b4912bb1595d16957b54" exitCode=0 Jul 09 22:41:44.922856 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:44.922520 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb" event={"ID":"0e728996-2f0e-4af9-a6ff-c01f09b0cc17","Type":"ContainerDied","Data":"14883d16a132c4785180136853cb37b624e26443bfd5b4912bb1595d16957b54"} Jul 09 22:41:46.052984 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:46.052954 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb" Jul 09 22:41:46.130106 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:46.130074 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e728996-2f0e-4af9-a6ff-c01f09b0cc17-util\") pod \"0e728996-2f0e-4af9-a6ff-c01f09b0cc17\" (UID: \"0e728996-2f0e-4af9-a6ff-c01f09b0cc17\") " Jul 09 22:41:46.130106 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:46.130114 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e728996-2f0e-4af9-a6ff-c01f09b0cc17-bundle\") pod \"0e728996-2f0e-4af9-a6ff-c01f09b0cc17\" (UID: \"0e728996-2f0e-4af9-a6ff-c01f09b0cc17\") " Jul 09 22:41:46.130370 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:46.130167 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltmbg\" (UniqueName: \"kubernetes.io/projected/0e728996-2f0e-4af9-a6ff-c01f09b0cc17-kube-api-access-ltmbg\") pod \"0e728996-2f0e-4af9-a6ff-c01f09b0cc17\" (UID: \"0e728996-2f0e-4af9-a6ff-c01f09b0cc17\") " Jul 09 22:41:46.130642 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:46.130610 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e728996-2f0e-4af9-a6ff-c01f09b0cc17-bundle" (OuterVolumeSpecName: "bundle") pod "0e728996-2f0e-4af9-a6ff-c01f09b0cc17" (UID: "0e728996-2f0e-4af9-a6ff-c01f09b0cc17"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:41:46.132413 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:46.132372 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e728996-2f0e-4af9-a6ff-c01f09b0cc17-kube-api-access-ltmbg" (OuterVolumeSpecName: "kube-api-access-ltmbg") pod "0e728996-2f0e-4af9-a6ff-c01f09b0cc17" (UID: "0e728996-2f0e-4af9-a6ff-c01f09b0cc17"). InnerVolumeSpecName "kube-api-access-ltmbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:41:46.135155 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:46.135134 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e728996-2f0e-4af9-a6ff-c01f09b0cc17-util" (OuterVolumeSpecName: "util") pod "0e728996-2f0e-4af9-a6ff-c01f09b0cc17" (UID: "0e728996-2f0e-4af9-a6ff-c01f09b0cc17"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:41:46.231141 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:46.231044 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltmbg\" (UniqueName: \"kubernetes.io/projected/0e728996-2f0e-4af9-a6ff-c01f09b0cc17-kube-api-access-ltmbg\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:41:46.231141 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:46.231077 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e728996-2f0e-4af9-a6ff-c01f09b0cc17-util\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:41:46.231141 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:46.231091 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e728996-2f0e-4af9-a6ff-c01f09b0cc17-bundle\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:41:46.933763 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:46.933729 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb" event={"ID":"0e728996-2f0e-4af9-a6ff-c01f09b0cc17","Type":"ContainerDied","Data":"8ea199da68dd640c295270c052d500ce8b6725bd8e4b559cbe46fd68e13ff38b"} Jul 09 22:41:46.933946 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:46.933778 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ea199da68dd640c295270c052d500ce8b6725bd8e4b559cbe46fd68e13ff38b" Jul 09 22:41:46.933946 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:46.933743 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftxvlb" Jul 09 22:41:50.414194 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:50.414150 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:41:50.514798 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:50.514761 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:41:52.243817 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.243784 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4m79d"] Jul 09 22:41:52.244279 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.244059 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e728996-2f0e-4af9-a6ff-c01f09b0cc17" containerName="pull" Jul 09 22:41:52.244279 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.244070 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e728996-2f0e-4af9-a6ff-c01f09b0cc17" containerName="pull" Jul 09 22:41:52.244279 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.244082 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e728996-2f0e-4af9-a6ff-c01f09b0cc17" containerName="util" Jul 09 22:41:52.244279 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.244088 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e728996-2f0e-4af9-a6ff-c01f09b0cc17" containerName="util" Jul 09 22:41:52.244279 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.244096 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e728996-2f0e-4af9-a6ff-c01f09b0cc17" containerName="extract" Jul 09 22:41:52.244279 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.244103 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e728996-2f0e-4af9-a6ff-c01f09b0cc17" containerName="extract" Jul 09 22:41:52.244279 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.244150 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e728996-2f0e-4af9-a6ff-c01f09b0cc17" containerName="extract" Jul 09 22:41:52.247100 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.247077 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4m79d" Jul 09 22:41:52.249938 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.249916 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 09 22:41:52.251166 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.251152 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 09 22:41:52.251166 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.251159 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-9fxgv\"" Jul 09 22:41:52.257676 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.257649 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4m79d"] Jul 09 22:41:52.279756 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.279715 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2767fc5c-f34b-4b29-89fc-fbb43a88df73-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4m79d\" (UID: \"2767fc5c-f34b-4b29-89fc-fbb43a88df73\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4m79d" Jul 09 22:41:52.279937 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.279868 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbq6q\" (UniqueName: \"kubernetes.io/projected/2767fc5c-f34b-4b29-89fc-fbb43a88df73-kube-api-access-pbq6q\") pod \"openshift-lws-operator-fd8ccff4c-4m79d\" (UID: \"2767fc5c-f34b-4b29-89fc-fbb43a88df73\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4m79d" Jul 09 22:41:52.381074 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.381032 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2767fc5c-f34b-4b29-89fc-fbb43a88df73-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4m79d\" (UID: \"2767fc5c-f34b-4b29-89fc-fbb43a88df73\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4m79d" Jul 09 22:41:52.381261 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.381214 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbq6q\" (UniqueName: \"kubernetes.io/projected/2767fc5c-f34b-4b29-89fc-fbb43a88df73-kube-api-access-pbq6q\") pod \"openshift-lws-operator-fd8ccff4c-4m79d\" (UID: \"2767fc5c-f34b-4b29-89fc-fbb43a88df73\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4m79d" Jul 09 22:41:52.381473 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.381455 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2767fc5c-f34b-4b29-89fc-fbb43a88df73-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4m79d\" (UID: \"2767fc5c-f34b-4b29-89fc-fbb43a88df73\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4m79d" Jul 09 22:41:52.389479 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.389441 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbq6q\" (UniqueName: \"kubernetes.io/projected/2767fc5c-f34b-4b29-89fc-fbb43a88df73-kube-api-access-pbq6q\") pod \"openshift-lws-operator-fd8ccff4c-4m79d\" (UID: \"2767fc5c-f34b-4b29-89fc-fbb43a88df73\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4m79d" Jul 09 22:41:52.557358 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.557278 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4m79d" Jul 09 22:41:52.696561 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.696529 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4m79d"] Jul 09 22:41:52.698990 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:41:52.698961 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2767fc5c_f34b_4b29_89fc_fbb43a88df73.slice/crio-693e16c26653385bbfea10ac4f4ef4ef014b841276ea9601d3ef9f6e576c3908 WatchSource:0}: Error finding container 693e16c26653385bbfea10ac4f4ef4ef014b841276ea9601d3ef9f6e576c3908: Status 404 returned error can't find the container with id 693e16c26653385bbfea10ac4f4ef4ef014b841276ea9601d3ef9f6e576c3908 Jul 09 22:41:52.959772 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:52.959682 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4m79d" event={"ID":"2767fc5c-f34b-4b29-89fc-fbb43a88df73","Type":"ContainerStarted","Data":"693e16c26653385bbfea10ac4f4ef4ef014b841276ea9601d3ef9f6e576c3908"} Jul 09 22:41:54.562215 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:54.562180 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:41:54.620382 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:54.620310 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:41:54.971181 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:54.971074 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4m79d" event={"ID":"2767fc5c-f34b-4b29-89fc-fbb43a88df73","Type":"ContainerStarted","Data":"28f7f12de5fb6cb2e8d7be2093cdb48bba6e9fe7ccd452d0156da699ae0a70a7"} Jul 09 22:41:54.987973 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:54.987922 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4m79d" podStartSLOduration=0.971710539 podStartE2EDuration="2.987903766s" totalImagesPullingTime="2.016193227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:41:54.987090356 +0000 UTC m=+539.528169163" watchObservedRunningTime="2026-07-09 22:41:54.987903766 +0000 UTC m=+539.528982562" Jul 09 22:41:55.670879 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:55.670842 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:41:55.771803 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:55.771770 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:41:57.632229 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:57.632185 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw"] Jul 09 22:41:57.635551 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:57.635522 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw" Jul 09 22:41:57.638328 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:57.638302 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 22:41:57.639559 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:57.639539 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 22:41:57.639559 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:57.639553 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-llmcv\"" Jul 09 22:41:57.643378 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:57.643351 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw"] Jul 09 22:41:57.723905 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:57.723862 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/64226a11-6d19-41fb-b531-88642dc12d5d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw\" (UID: \"64226a11-6d19-41fb-b531-88642dc12d5d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw" Jul 09 22:41:57.724076 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:57.723915 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj8bh\" (UniqueName: \"kubernetes.io/projected/64226a11-6d19-41fb-b531-88642dc12d5d-kube-api-access-dj8bh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw\" (UID: \"64226a11-6d19-41fb-b531-88642dc12d5d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw" Jul 09 22:41:57.724076 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:57.723943 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/64226a11-6d19-41fb-b531-88642dc12d5d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw\" (UID: \"64226a11-6d19-41fb-b531-88642dc12d5d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw" Jul 09 22:41:57.824720 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:57.824681 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/64226a11-6d19-41fb-b531-88642dc12d5d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw\" (UID: \"64226a11-6d19-41fb-b531-88642dc12d5d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw" Jul 09 22:41:57.824898 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:57.824731 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj8bh\" (UniqueName: \"kubernetes.io/projected/64226a11-6d19-41fb-b531-88642dc12d5d-kube-api-access-dj8bh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw\" (UID: \"64226a11-6d19-41fb-b531-88642dc12d5d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw" Jul 09 22:41:57.824898 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:57.824762 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/64226a11-6d19-41fb-b531-88642dc12d5d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw\" (UID: \"64226a11-6d19-41fb-b531-88642dc12d5d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw" Jul 09 22:41:57.825177 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:57.825146 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/64226a11-6d19-41fb-b531-88642dc12d5d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw\" (UID: \"64226a11-6d19-41fb-b531-88642dc12d5d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw" Jul 09 22:41:57.825177 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:57.825160 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/64226a11-6d19-41fb-b531-88642dc12d5d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw\" (UID: \"64226a11-6d19-41fb-b531-88642dc12d5d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw" Jul 09 22:41:57.836216 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:57.836188 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj8bh\" (UniqueName: \"kubernetes.io/projected/64226a11-6d19-41fb-b531-88642dc12d5d-kube-api-access-dj8bh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw\" (UID: \"64226a11-6d19-41fb-b531-88642dc12d5d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw" Jul 09 22:41:57.863073 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:57.863027 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:41:57.945673 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:57.945586 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw" Jul 09 22:41:58.084455 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:58.084430 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw"] Jul 09 22:41:58.087097 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:41:58.087068 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64226a11_6d19_41fb_b531_88642dc12d5d.slice/crio-5e108b229f170688bfc198fb3751f791df23a829d5dd28693186d2ec89650f35 WatchSource:0}: Error finding container 5e108b229f170688bfc198fb3751f791df23a829d5dd28693186d2ec89650f35: Status 404 returned error can't find the container with id 5e108b229f170688bfc198fb3751f791df23a829d5dd28693186d2ec89650f35 Jul 09 22:41:58.990111 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:58.990076 2568 generic.go:356] "Generic (PLEG): container finished" podID="64226a11-6d19-41fb-b531-88642dc12d5d" containerID="05e0c9a4a2ad30efed2eae6e92ffcea4f58afdeaf860e24f86adb596ce69ebb2" exitCode=0 Jul 09 22:41:58.990650 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:58.990169 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw" event={"ID":"64226a11-6d19-41fb-b531-88642dc12d5d","Type":"ContainerDied","Data":"05e0c9a4a2ad30efed2eae6e92ffcea4f58afdeaf860e24f86adb596ce69ebb2"} Jul 09 22:41:58.990650 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:58.990222 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw" event={"ID":"64226a11-6d19-41fb-b531-88642dc12d5d","Type":"ContainerStarted","Data":"5e108b229f170688bfc198fb3751f791df23a829d5dd28693186d2ec89650f35"} Jul 09 22:41:59.996552 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:59.996439 2568 generic.go:356] "Generic (PLEG): container finished" podID="64226a11-6d19-41fb-b531-88642dc12d5d" containerID="92094b6e0d67a3d6f7eb6ba6c500f7b02edc9f2d31b760db9d20915a80791a77" exitCode=0 Jul 09 22:41:59.996552 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:41:59.996533 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw" event={"ID":"64226a11-6d19-41fb-b531-88642dc12d5d","Type":"ContainerDied","Data":"92094b6e0d67a3d6f7eb6ba6c500f7b02edc9f2d31b760db9d20915a80791a77"} Jul 09 22:42:01.005977 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:01.005937 2568 generic.go:356] "Generic (PLEG): container finished" podID="64226a11-6d19-41fb-b531-88642dc12d5d" containerID="20cd71d2f0b990f7aa28007e83bf3c41aaa4c4b93faceb82868fec1230bfbf89" exitCode=0 Jul 09 22:42:01.006358 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:01.006021 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw" event={"ID":"64226a11-6d19-41fb-b531-88642dc12d5d","Type":"ContainerDied","Data":"20cd71d2f0b990f7aa28007e83bf3c41aaa4c4b93faceb82868fec1230bfbf89"} Jul 09 22:42:02.137332 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:02.137301 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw" Jul 09 22:42:02.259329 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:02.259286 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/64226a11-6d19-41fb-b531-88642dc12d5d-bundle\") pod \"64226a11-6d19-41fb-b531-88642dc12d5d\" (UID: \"64226a11-6d19-41fb-b531-88642dc12d5d\") " Jul 09 22:42:02.259542 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:02.259354 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/64226a11-6d19-41fb-b531-88642dc12d5d-util\") pod \"64226a11-6d19-41fb-b531-88642dc12d5d\" (UID: \"64226a11-6d19-41fb-b531-88642dc12d5d\") " Jul 09 22:42:02.259542 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:02.259381 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dj8bh\" (UniqueName: \"kubernetes.io/projected/64226a11-6d19-41fb-b531-88642dc12d5d-kube-api-access-dj8bh\") pod \"64226a11-6d19-41fb-b531-88642dc12d5d\" (UID: \"64226a11-6d19-41fb-b531-88642dc12d5d\") " Jul 09 22:42:02.260125 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:02.260098 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64226a11-6d19-41fb-b531-88642dc12d5d-bundle" (OuterVolumeSpecName: "bundle") pod "64226a11-6d19-41fb-b531-88642dc12d5d" (UID: "64226a11-6d19-41fb-b531-88642dc12d5d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:42:02.261601 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:02.261527 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64226a11-6d19-41fb-b531-88642dc12d5d-kube-api-access-dj8bh" (OuterVolumeSpecName: "kube-api-access-dj8bh") pod "64226a11-6d19-41fb-b531-88642dc12d5d" (UID: "64226a11-6d19-41fb-b531-88642dc12d5d"). InnerVolumeSpecName "kube-api-access-dj8bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:42:02.265376 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:02.265343 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64226a11-6d19-41fb-b531-88642dc12d5d-util" (OuterVolumeSpecName: "util") pod "64226a11-6d19-41fb-b531-88642dc12d5d" (UID: "64226a11-6d19-41fb-b531-88642dc12d5d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:42:02.360843 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:02.360799 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/64226a11-6d19-41fb-b531-88642dc12d5d-bundle\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:42:02.360843 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:02.360829 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/64226a11-6d19-41fb-b531-88642dc12d5d-util\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:42:02.360843 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:02.360838 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj8bh\" (UniqueName: \"kubernetes.io/projected/64226a11-6d19-41fb-b531-88642dc12d5d-kube-api-access-dj8bh\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:42:03.016020 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:03.015987 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw" event={"ID":"64226a11-6d19-41fb-b531-88642dc12d5d","Type":"ContainerDied","Data":"5e108b229f170688bfc198fb3751f791df23a829d5dd28693186d2ec89650f35"} Jul 09 22:42:03.016196 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:03.016030 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft8qsw" Jul 09 22:42:03.016196 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:03.016033 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e108b229f170688bfc198fb3751f791df23a829d5dd28693186d2ec89650f35" Jul 09 22:42:06.102788 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.102752 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:06.162987 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.162954 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc"] Jul 09 22:42:06.163278 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.163266 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64226a11-6d19-41fb-b531-88642dc12d5d" containerName="extract" Jul 09 22:42:06.163319 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.163279 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="64226a11-6d19-41fb-b531-88642dc12d5d" containerName="extract" Jul 09 22:42:06.163319 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.163292 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64226a11-6d19-41fb-b531-88642dc12d5d" containerName="util" Jul 09 22:42:06.163319 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.163297 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="64226a11-6d19-41fb-b531-88642dc12d5d" containerName="util" Jul 09 22:42:06.163319 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.163308 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64226a11-6d19-41fb-b531-88642dc12d5d" containerName="pull" Jul 09 22:42:06.163319 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.163312 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="64226a11-6d19-41fb-b531-88642dc12d5d" containerName="pull" Jul 09 22:42:06.163466 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.163356 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="64226a11-6d19-41fb-b531-88642dc12d5d" containerName="extract" Jul 09 22:42:06.165361 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.165343 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc" Jul 09 22:42:06.169532 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.169508 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 09 22:42:06.169696 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.169535 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 09 22:42:06.169696 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.169535 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 09 22:42:06.169816 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.169750 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-bhmjm\"" Jul 09 22:42:06.175190 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.175165 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc"] Jul 09 22:42:06.193225 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.193187 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:06.291186 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.291145 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/da80d4dd-0f06-4e5a-be7c-7f7f95cfed92-metrics-cert\") pod \"lws-controller-manager-5f8b5887bc-jq4jc\" (UID: \"da80d4dd-0f06-4e5a-be7c-7f7f95cfed92\") " pod="openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc" Jul 09 22:42:06.291366 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.291208 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzlsb\" (UniqueName: \"kubernetes.io/projected/da80d4dd-0f06-4e5a-be7c-7f7f95cfed92-kube-api-access-dzlsb\") pod \"lws-controller-manager-5f8b5887bc-jq4jc\" (UID: \"da80d4dd-0f06-4e5a-be7c-7f7f95cfed92\") " pod="openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc" Jul 09 22:42:06.291366 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.291292 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da80d4dd-0f06-4e5a-be7c-7f7f95cfed92-cert\") pod \"lws-controller-manager-5f8b5887bc-jq4jc\" (UID: \"da80d4dd-0f06-4e5a-be7c-7f7f95cfed92\") " pod="openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc" Jul 09 22:42:06.291366 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.291345 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da80d4dd-0f06-4e5a-be7c-7f7f95cfed92-manager-config\") pod \"lws-controller-manager-5f8b5887bc-jq4jc\" (UID: \"da80d4dd-0f06-4e5a-be7c-7f7f95cfed92\") " pod="openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc" Jul 09 22:42:06.392240 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.392149 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da80d4dd-0f06-4e5a-be7c-7f7f95cfed92-manager-config\") pod \"lws-controller-manager-5f8b5887bc-jq4jc\" (UID: \"da80d4dd-0f06-4e5a-be7c-7f7f95cfed92\") " pod="openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc" Jul 09 22:42:06.392240 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.392200 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/da80d4dd-0f06-4e5a-be7c-7f7f95cfed92-metrics-cert\") pod \"lws-controller-manager-5f8b5887bc-jq4jc\" (UID: \"da80d4dd-0f06-4e5a-be7c-7f7f95cfed92\") " pod="openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc" Jul 09 22:42:06.392240 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.392233 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzlsb\" (UniqueName: \"kubernetes.io/projected/da80d4dd-0f06-4e5a-be7c-7f7f95cfed92-kube-api-access-dzlsb\") pod \"lws-controller-manager-5f8b5887bc-jq4jc\" (UID: \"da80d4dd-0f06-4e5a-be7c-7f7f95cfed92\") " pod="openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc" Jul 09 22:42:06.392550 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.392266 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da80d4dd-0f06-4e5a-be7c-7f7f95cfed92-cert\") pod \"lws-controller-manager-5f8b5887bc-jq4jc\" (UID: \"da80d4dd-0f06-4e5a-be7c-7f7f95cfed92\") " pod="openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc" Jul 09 22:42:06.392850 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.392826 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/da80d4dd-0f06-4e5a-be7c-7f7f95cfed92-manager-config\") pod \"lws-controller-manager-5f8b5887bc-jq4jc\" (UID: \"da80d4dd-0f06-4e5a-be7c-7f7f95cfed92\") " pod="openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc" Jul 09 22:42:06.394730 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.394706 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da80d4dd-0f06-4e5a-be7c-7f7f95cfed92-cert\") pod \"lws-controller-manager-5f8b5887bc-jq4jc\" (UID: \"da80d4dd-0f06-4e5a-be7c-7f7f95cfed92\") " pod="openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc" Jul 09 22:42:06.394846 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.394829 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/da80d4dd-0f06-4e5a-be7c-7f7f95cfed92-metrics-cert\") pod \"lws-controller-manager-5f8b5887bc-jq4jc\" (UID: \"da80d4dd-0f06-4e5a-be7c-7f7f95cfed92\") " pod="openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc" Jul 09 22:42:06.401434 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.401413 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzlsb\" (UniqueName: \"kubernetes.io/projected/da80d4dd-0f06-4e5a-be7c-7f7f95cfed92-kube-api-access-dzlsb\") pod \"lws-controller-manager-5f8b5887bc-jq4jc\" (UID: \"da80d4dd-0f06-4e5a-be7c-7f7f95cfed92\") " pod="openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc" Jul 09 22:42:06.475164 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.475120 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc" Jul 09 22:42:06.609357 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:06.609333 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc"] Jul 09 22:42:06.611421 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:42:06.611394 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda80d4dd_0f06_4e5a_be7c_7f7f95cfed92.slice/crio-e6bcfecebe8328b90cd5fa6e3b19f77df6c32165f557baf9169efc5c4c3298ef WatchSource:0}: Error finding container e6bcfecebe8328b90cd5fa6e3b19f77df6c32165f557baf9169efc5c4c3298ef: Status 404 returned error can't find the container with id e6bcfecebe8328b90cd5fa6e3b19f77df6c32165f557baf9169efc5c4c3298ef Jul 09 22:42:07.034835 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:07.034803 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc" event={"ID":"da80d4dd-0f06-4e5a-be7c-7f7f95cfed92","Type":"ContainerStarted","Data":"e6bcfecebe8328b90cd5fa6e3b19f77df6c32165f557baf9169efc5c4c3298ef"} Jul 09 22:42:09.046734 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:09.046694 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc" event={"ID":"da80d4dd-0f06-4e5a-be7c-7f7f95cfed92","Type":"ContainerStarted","Data":"fd7d0e3b442ff63f9d11e43717c0130f08cc1c8fb78000bdb5423e1b6b325459"} Jul 09 22:42:09.047135 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:09.046824 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc" Jul 09 22:42:09.069951 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:09.069899 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc" podStartSLOduration=1.457569839 podStartE2EDuration="3.069882208s" totalImagesPullingTime="1.612312369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:42:09.067475003 +0000 UTC m=+553.608553798" watchObservedRunningTime="2026-07-09 22:42:09.069882208 +0000 UTC m=+553.610961002" Jul 09 22:42:13.377471 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:13.377433 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:13.479302 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:13.479259 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:13.978038 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:13.978002 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:14.282081 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.281990 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:14.389035 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.389002 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:14.443167 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.443126 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8"] Jul 09 22:42:14.446892 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.446871 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" Jul 09 22:42:14.450628 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.450604 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 22:42:14.450761 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.450602 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 22:42:14.450938 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.450925 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 22:42:14.453195 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.453174 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 22:42:14.456176 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.456157 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k9dxk\"" Jul 09 22:42:14.469314 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.469285 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8"] Jul 09 22:42:14.565848 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.565811 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc5p2\" (UniqueName: \"kubernetes.io/projected/cdae79af-b625-4446-ae94-0b79e35df8fa-kube-api-access-sc5p2\") pod \"opendatahub-operator-controller-manager-54dbdc554f-rk8z8\" (UID: \"cdae79af-b625-4446-ae94-0b79e35df8fa\") " pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" Jul 09 22:42:14.566029 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.565856 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cdae79af-b625-4446-ae94-0b79e35df8fa-manifests\") pod \"opendatahub-operator-controller-manager-54dbdc554f-rk8z8\" (UID: \"cdae79af-b625-4446-ae94-0b79e35df8fa\") " pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" Jul 09 22:42:14.566029 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.565929 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cdae79af-b625-4446-ae94-0b79e35df8fa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54dbdc554f-rk8z8\" (UID: \"cdae79af-b625-4446-ae94-0b79e35df8fa\") " pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" Jul 09 22:42:14.566029 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.565964 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cdae79af-b625-4446-ae94-0b79e35df8fa-webhook-cert\") pod \"opendatahub-operator-controller-manager-54dbdc554f-rk8z8\" (UID: \"cdae79af-b625-4446-ae94-0b79e35df8fa\") " pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" Jul 09 22:42:14.566029 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.565997 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdae79af-b625-4446-ae94-0b79e35df8fa-tmp\") pod \"opendatahub-operator-controller-manager-54dbdc554f-rk8z8\" (UID: \"cdae79af-b625-4446-ae94-0b79e35df8fa\") " pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" Jul 09 22:42:14.607828 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.607791 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:14.666641 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.666609 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc5p2\" (UniqueName: \"kubernetes.io/projected/cdae79af-b625-4446-ae94-0b79e35df8fa-kube-api-access-sc5p2\") pod \"opendatahub-operator-controller-manager-54dbdc554f-rk8z8\" (UID: \"cdae79af-b625-4446-ae94-0b79e35df8fa\") " pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" Jul 09 22:42:14.666641 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.666648 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cdae79af-b625-4446-ae94-0b79e35df8fa-manifests\") pod \"opendatahub-operator-controller-manager-54dbdc554f-rk8z8\" (UID: \"cdae79af-b625-4446-ae94-0b79e35df8fa\") " pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" Jul 09 22:42:14.666971 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.666679 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cdae79af-b625-4446-ae94-0b79e35df8fa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54dbdc554f-rk8z8\" (UID: \"cdae79af-b625-4446-ae94-0b79e35df8fa\") " pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" Jul 09 22:42:14.666971 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.666700 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cdae79af-b625-4446-ae94-0b79e35df8fa-webhook-cert\") pod \"opendatahub-operator-controller-manager-54dbdc554f-rk8z8\" (UID: \"cdae79af-b625-4446-ae94-0b79e35df8fa\") " pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" Jul 09 22:42:14.666971 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.666851 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdae79af-b625-4446-ae94-0b79e35df8fa-tmp\") pod \"opendatahub-operator-controller-manager-54dbdc554f-rk8z8\" (UID: \"cdae79af-b625-4446-ae94-0b79e35df8fa\") " pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" Jul 09 22:42:14.667102 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.667081 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cdae79af-b625-4446-ae94-0b79e35df8fa-manifests\") pod \"opendatahub-operator-controller-manager-54dbdc554f-rk8z8\" (UID: \"cdae79af-b625-4446-ae94-0b79e35df8fa\") " pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" Jul 09 22:42:14.669153 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.669126 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdae79af-b625-4446-ae94-0b79e35df8fa-tmp\") pod \"opendatahub-operator-controller-manager-54dbdc554f-rk8z8\" (UID: \"cdae79af-b625-4446-ae94-0b79e35df8fa\") " pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" Jul 09 22:42:14.669322 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.669300 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cdae79af-b625-4446-ae94-0b79e35df8fa-webhook-cert\") pod \"opendatahub-operator-controller-manager-54dbdc554f-rk8z8\" (UID: \"cdae79af-b625-4446-ae94-0b79e35df8fa\") " pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" Jul 09 22:42:14.669379 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.669351 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cdae79af-b625-4446-ae94-0b79e35df8fa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54dbdc554f-rk8z8\" (UID: \"cdae79af-b625-4446-ae94-0b79e35df8fa\") " pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" Jul 09 22:42:14.682649 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.682617 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc5p2\" (UniqueName: \"kubernetes.io/projected/cdae79af-b625-4446-ae94-0b79e35df8fa-kube-api-access-sc5p2\") pod \"opendatahub-operator-controller-manager-54dbdc554f-rk8z8\" (UID: \"cdae79af-b625-4446-ae94-0b79e35df8fa\") " pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" Jul 09 22:42:14.707764 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.707723 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:14.758147 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.758107 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" Jul 09 22:42:14.808332 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.808225 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:14.898559 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.898536 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8"] Jul 09 22:42:14.901141 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.901115 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:14.901284 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:42:14.901178 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdae79af_b625_4446_ae94_0b79e35df8fa.slice/crio-9f09f92578aa8ef0e0f63acdfeee87dbe329c4778edf2586f6736528eb9fdeb8 WatchSource:0}: Error finding container 9f09f92578aa8ef0e0f63acdfeee87dbe329c4778edf2586f6736528eb9fdeb8: Status 404 returned error can't find the container with id 9f09f92578aa8ef0e0f63acdfeee87dbe329c4778edf2586f6736528eb9fdeb8 Jul 09 22:42:14.969756 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:14.969710 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:15.070107 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:15.070075 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:15.075998 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:15.075970 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" event={"ID":"cdae79af-b625-4446-ae94-0b79e35df8fa","Type":"ContainerStarted","Data":"9f09f92578aa8ef0e0f63acdfeee87dbe329c4778edf2586f6736528eb9fdeb8"} Jul 09 22:42:15.175550 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:15.175450 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:18.092738 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:18.092648 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" event={"ID":"cdae79af-b625-4446-ae94-0b79e35df8fa","Type":"ContainerStarted","Data":"891a388f5c101e85617a9f03b8f868517e3299dddb9f096d7c918f846e595b89"} Jul 09 22:42:19.099189 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:19.099152 2568 generic.go:356] "Generic (PLEG): container finished" podID="cdae79af-b625-4446-ae94-0b79e35df8fa" containerID="891a388f5c101e85617a9f03b8f868517e3299dddb9f096d7c918f846e595b89" exitCode=0 Jul 09 22:42:19.099704 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:19.099218 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" event={"ID":"cdae79af-b625-4446-ae94-0b79e35df8fa","Type":"ContainerDied","Data":"891a388f5c101e85617a9f03b8f868517e3299dddb9f096d7c918f846e595b89"} Jul 09 22:42:20.055955 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:20.055922 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5f8b5887bc-jq4jc" Jul 09 22:42:20.106737 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:20.106698 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" event={"ID":"cdae79af-b625-4446-ae94-0b79e35df8fa","Type":"ContainerStarted","Data":"a108b191a030f5ee8a2905ce7900e23e93bd1eec7724ed23547309b06c6db393"} Jul 09 22:42:20.107144 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:20.106811 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" Jul 09 22:42:20.131221 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:20.131172 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" podStartSLOduration=1.836302919 podStartE2EDuration="6.131154286s" totalImagesPullingTime="3.209648264s" totalInitContainerRuntime="1.085203103s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:42:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:42:20.129770017 +0000 UTC m=+564.670848831" watchObservedRunningTime="2026-07-09 22:42:20.131154286 +0000 UTC m=+564.672233079" Jul 09 22:42:26.398781 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:26.398743 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:26.827801 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:26.827767 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:26.929233 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:26.929195 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:31.118345 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:31.118315 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-54dbdc554f-rk8z8" Jul 09 22:42:32.915415 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:32.915373 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:34.436248 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:34.436211 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:34.722246 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:34.722078 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:34.834960 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:34.834932 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:34.913433 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:34.913394 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:35.019041 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:35.018946 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:35.957132 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:35.957093 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:36.896972 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:36.896935 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:40.733346 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:40.733315 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:40.836214 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:40.836177 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:41.035252 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.035164 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8"] Jul 09 22:42:41.043569 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.043541 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.046412 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.046391 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 22:42:41.046621 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.046599 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-7fsg4\"" Jul 09 22:42:41.046694 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.046649 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 22:42:41.046758 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.046721 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 22:42:41.052485 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.052462 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8"] Jul 09 22:42:41.076509 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.076457 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:41.088649 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.088620 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.088772 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.088661 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.088772 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.088680 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.088772 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.088726 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.088772 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.088750 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.088772 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.088765 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptgst\" (UniqueName: \"kubernetes.io/projected/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-kube-api-access-ptgst\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.088971 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.088791 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.088971 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.088846 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.088971 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.088888 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.176566 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.176529 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:41.189562 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.189538 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.189704 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.189584 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.189704 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.189614 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.189704 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.189657 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.189704 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.189673 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.189704 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.189694 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.189956 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.189795 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.189956 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.189838 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptgst\" (UniqueName: \"kubernetes.io/projected/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-kube-api-access-ptgst\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.189956 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.189888 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.190103 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.189984 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.190103 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.190040 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.190289 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.190253 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.190420 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.190384 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.190625 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.190604 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.192175 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.192153 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.192403 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.192388 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.208196 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.208148 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.212180 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.212158 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptgst\" (UniqueName: \"kubernetes.io/projected/c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f-kube-api-access-ptgst\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lz5v8\" (UID: \"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.283562 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.283524 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:41.354952 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.354921 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:41.510563 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:41.510542 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8"] Jul 09 22:42:41.512653 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:42:41.512624 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c71d63_b3ab_40d7_a95e_6de7a9e81a4f.slice/crio-49ca2b9c07971a7f228a4f97e63be97c99563f69a10cfd083a51084964058033 WatchSource:0}: Error finding container 49ca2b9c07971a7f228a4f97e63be97c99563f69a10cfd083a51084964058033: Status 404 returned error can't find the container with id 49ca2b9c07971a7f228a4f97e63be97c99563f69a10cfd083a51084964058033 Jul 09 22:42:42.213660 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:42.213591 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" event={"ID":"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f","Type":"ContainerStarted","Data":"49ca2b9c07971a7f228a4f97e63be97c99563f69a10cfd083a51084964058033"} Jul 09 22:42:43.854917 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:43.854884 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx"] Jul 09 22:42:43.862629 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:43.862596 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:43.872514 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:43.869815 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ldsds\"" Jul 09 22:42:43.872514 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:43.870047 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 09 22:42:43.872514 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:43.870220 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 09 22:42:43.872514 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:43.870370 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 09 22:42:43.877817 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:43.877787 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx"] Jul 09 22:42:43.910730 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:43.910695 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49e1048-733d-41a3-b184-48173c9132bb-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:43.910929 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:43.910775 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c49e1048-733d-41a3-b184-48173c9132bb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:43.910929 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:43.910804 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49e1048-733d-41a3-b184-48173c9132bb-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:43.910929 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:43.910830 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c49e1048-733d-41a3-b184-48173c9132bb-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:43.910929 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:43.910854 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c49e1048-733d-41a3-b184-48173c9132bb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:43.910929 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:43.910896 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jbnx\" (UniqueName: \"kubernetes.io/projected/c49e1048-733d-41a3-b184-48173c9132bb-kube-api-access-8jbnx\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:43.911120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:43.910987 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c49e1048-733d-41a3-b184-48173c9132bb-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:44.011812 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:44.011778 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jbnx\" (UniqueName: \"kubernetes.io/projected/c49e1048-733d-41a3-b184-48173c9132bb-kube-api-access-8jbnx\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:44.012008 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:44.011825 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c49e1048-733d-41a3-b184-48173c9132bb-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:44.012008 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:44.011856 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49e1048-733d-41a3-b184-48173c9132bb-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:44.012008 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:44.011905 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c49e1048-733d-41a3-b184-48173c9132bb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:44.012188 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:44.012054 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49e1048-733d-41a3-b184-48173c9132bb-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:44.012188 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:44.012143 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c49e1048-733d-41a3-b184-48173c9132bb-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:44.012188 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:44.012180 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c49e1048-733d-41a3-b184-48173c9132bb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:44.013029 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:44.012998 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c49e1048-733d-41a3-b184-48173c9132bb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:44.014661 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:44.014625 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c49e1048-733d-41a3-b184-48173c9132bb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:44.014792 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:44.014639 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c49e1048-733d-41a3-b184-48173c9132bb-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:44.014792 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:44.014777 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49e1048-733d-41a3-b184-48173c9132bb-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:44.014937 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:44.014918 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c49e1048-733d-41a3-b184-48173c9132bb-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:44.021171 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:44.021129 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49e1048-733d-41a3-b184-48173c9132bb-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:44.021332 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:44.021269 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jbnx\" (UniqueName: \"kubernetes.io/projected/c49e1048-733d-41a3-b184-48173c9132bb-kube-api-access-8jbnx\") pod \"istiod-openshift-gateway-94bb8fbfd-j48wx\" (UID: \"c49e1048-733d-41a3-b184-48173c9132bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:44.185730 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:44.185651 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:44.216872 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:44.216834 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:42:44.216977 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:44.216912 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:42:44.216977 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:44.216939 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:42:44.358827 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:44.358738 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx"] Jul 09 22:42:44.361625 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:42:44.361597 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49e1048_733d_41a3_b184_48173c9132bb.slice/crio-83e909c38383c36cf4bea20c5c3a0210964752a8d310a4ac9ab90cd34efe3f89 WatchSource:0}: Error finding container 83e909c38383c36cf4bea20c5c3a0210964752a8d310a4ac9ab90cd34efe3f89: Status 404 returned error can't find the container with id 83e909c38383c36cf4bea20c5c3a0210964752a8d310a4ac9ab90cd34efe3f89 Jul 09 22:42:45.028510 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:45.028463 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:45.129532 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:45.129478 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:45.234678 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:45.234637 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" event={"ID":"c49e1048-733d-41a3-b184-48173c9132bb","Type":"ContainerStarted","Data":"83e909c38383c36cf4bea20c5c3a0210964752a8d310a4ac9ab90cd34efe3f89"} Jul 09 22:42:45.236446 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:45.236411 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" event={"ID":"c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f","Type":"ContainerStarted","Data":"10de9a99f9a3517f02cae5e7eb9c6ab71ae3e7e317aff6cdb125662f277b5a0f"} Jul 09 22:42:45.268815 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:45.268768 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" podStartSLOduration=1.566746414 podStartE2EDuration="4.268747714s" totalImagesPullingTime="2.7020013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:42:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:42:45.266373976 +0000 UTC m=+589.807452797" watchObservedRunningTime="2026-07-09 22:42:45.268747714 +0000 UTC m=+589.809826507" Jul 09 22:42:45.338855 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:45.338820 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:45.355651 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:45.355619 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:45.361607 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:45.361579 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:45.434056 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:45.433968 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:46.241208 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:46.241174 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:46.242374 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:46.242348 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lz5v8" Jul 09 22:42:46.401634 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:46.401592 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:46.494625 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:46.494543 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:46.529651 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:46.529602 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:42:46.529806 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:46.529703 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:42:46.595112 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:46.595071 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:47.247833 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:47.247765 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" event={"ID":"c49e1048-733d-41a3-b184-48173c9132bb","Type":"ContainerStarted","Data":"e0c589d246af6e3ed71750e1c1d86a04e2fbd8ccc8df3873774c817511de8ae3"} Jul 09 22:42:47.274742 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:47.274669 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" podStartSLOduration=2.109373562 podStartE2EDuration="4.274649891s" totalImagesPullingTime="2.165276329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:42:47.273610764 +0000 UTC m=+591.814689583" watchObservedRunningTime="2026-07-09 22:42:47.274649891 +0000 UTC m=+591.815728686" Jul 09 22:42:48.251798 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:48.251766 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:48.253376 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:48.253341 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j48wx" Jul 09 22:42:48.639585 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:48.639550 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:42:48.749237 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:42:48.749198 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:43:00.787998 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:00.787962 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:43:00.886695 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:00.886647 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:43:20.657428 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:20.657346 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:43:34.385585 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:34.385547 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:43:34.487927 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:34.487880 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:43:34.585334 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:34.585301 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:43:35.303550 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:35.303511 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:43:35.511665 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:35.511635 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:43:50.291677 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:50.291640 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb"] Jul 09 22:43:50.295048 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:50.295029 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb" Jul 09 22:43:50.297997 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:50.297978 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 22:43:50.299372 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:50.299354 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 22:43:50.299372 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:50.299365 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-x6swm\"" Jul 09 22:43:50.305610 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:50.305588 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb"] Jul 09 22:43:50.389129 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:50.389085 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbdnq\" (UniqueName: \"kubernetes.io/projected/f856756d-c7e6-4102-8598-cdcf4cfb62fe-kube-api-access-sbdnq\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb\" (UID: \"f856756d-c7e6-4102-8598-cdcf4cfb62fe\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb" Jul 09 22:43:50.389334 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:50.389146 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f856756d-c7e6-4102-8598-cdcf4cfb62fe-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb\" (UID: \"f856756d-c7e6-4102-8598-cdcf4cfb62fe\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb" Jul 09 22:43:50.389334 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:50.389260 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f856756d-c7e6-4102-8598-cdcf4cfb62fe-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb\" (UID: \"f856756d-c7e6-4102-8598-cdcf4cfb62fe\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb" Jul 09 22:43:50.490543 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:50.490475 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f856756d-c7e6-4102-8598-cdcf4cfb62fe-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb\" (UID: \"f856756d-c7e6-4102-8598-cdcf4cfb62fe\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb" Jul 09 22:43:50.490734 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:50.490588 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbdnq\" (UniqueName: \"kubernetes.io/projected/f856756d-c7e6-4102-8598-cdcf4cfb62fe-kube-api-access-sbdnq\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb\" (UID: \"f856756d-c7e6-4102-8598-cdcf4cfb62fe\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb" Jul 09 22:43:50.490734 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:50.490631 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f856756d-c7e6-4102-8598-cdcf4cfb62fe-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb\" (UID: \"f856756d-c7e6-4102-8598-cdcf4cfb62fe\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb" Jul 09 22:43:50.490925 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:50.490902 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f856756d-c7e6-4102-8598-cdcf4cfb62fe-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb\" (UID: \"f856756d-c7e6-4102-8598-cdcf4cfb62fe\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb" Jul 09 22:43:50.490988 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:50.490937 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f856756d-c7e6-4102-8598-cdcf4cfb62fe-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb\" (UID: \"f856756d-c7e6-4102-8598-cdcf4cfb62fe\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb" Jul 09 22:43:50.500530 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:50.500484 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbdnq\" (UniqueName: \"kubernetes.io/projected/f856756d-c7e6-4102-8598-cdcf4cfb62fe-kube-api-access-sbdnq\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb\" (UID: \"f856756d-c7e6-4102-8598-cdcf4cfb62fe\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb" Jul 09 22:43:50.605981 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:50.605946 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb" Jul 09 22:43:50.739606 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:50.739579 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb"] Jul 09 22:43:50.741576 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:43:50.741549 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf856756d_c7e6_4102_8598_cdcf4cfb62fe.slice/crio-26df126ee6f300c7c3833fce43145ead0386fed91ea105ae1502841f3aab5d92 WatchSource:0}: Error finding container 26df126ee6f300c7c3833fce43145ead0386fed91ea105ae1502841f3aab5d92: Status 404 returned error can't find the container with id 26df126ee6f300c7c3833fce43145ead0386fed91ea105ae1502841f3aab5d92 Jul 09 22:43:51.084777 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.084743 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k"] Jul 09 22:43:51.087960 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.087942 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k" Jul 09 22:43:51.098603 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.098576 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k"] Jul 09 22:43:51.197157 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.197117 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b696f428-8899-4dfb-91ee-4ecf429ba33a-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k\" (UID: \"b696f428-8899-4dfb-91ee-4ecf429ba33a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k" Jul 09 22:43:51.197157 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.197167 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b696f428-8899-4dfb-91ee-4ecf429ba33a-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k\" (UID: \"b696f428-8899-4dfb-91ee-4ecf429ba33a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k" Jul 09 22:43:51.197407 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.197284 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5v4d\" (UniqueName: \"kubernetes.io/projected/b696f428-8899-4dfb-91ee-4ecf429ba33a-kube-api-access-p5v4d\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k\" (UID: \"b696f428-8899-4dfb-91ee-4ecf429ba33a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k" Jul 09 22:43:51.297720 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.297683 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5v4d\" (UniqueName: \"kubernetes.io/projected/b696f428-8899-4dfb-91ee-4ecf429ba33a-kube-api-access-p5v4d\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k\" (UID: \"b696f428-8899-4dfb-91ee-4ecf429ba33a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k" Jul 09 22:43:51.298105 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.297755 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b696f428-8899-4dfb-91ee-4ecf429ba33a-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k\" (UID: \"b696f428-8899-4dfb-91ee-4ecf429ba33a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k" Jul 09 22:43:51.298105 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.297774 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b696f428-8899-4dfb-91ee-4ecf429ba33a-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k\" (UID: \"b696f428-8899-4dfb-91ee-4ecf429ba33a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k" Jul 09 22:43:51.298177 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.298131 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b696f428-8899-4dfb-91ee-4ecf429ba33a-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k\" (UID: \"b696f428-8899-4dfb-91ee-4ecf429ba33a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k" Jul 09 22:43:51.298215 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.298202 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b696f428-8899-4dfb-91ee-4ecf429ba33a-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k\" (UID: \"b696f428-8899-4dfb-91ee-4ecf429ba33a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k" Jul 09 22:43:51.306982 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.306955 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5v4d\" (UniqueName: \"kubernetes.io/projected/b696f428-8899-4dfb-91ee-4ecf429ba33a-kube-api-access-p5v4d\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k\" (UID: \"b696f428-8899-4dfb-91ee-4ecf429ba33a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k" Jul 09 22:43:51.397149 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.397065 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k" Jul 09 22:43:51.541310 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.541117 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k"] Jul 09 22:43:51.543429 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:43:51.543402 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb696f428_8899_4dfb_91ee_4ecf429ba33a.slice/crio-8f7021ffdf799acd41c02645cce626a2214746439f1885b05fb975a7286f5fda WatchSource:0}: Error finding container 8f7021ffdf799acd41c02645cce626a2214746439f1885b05fb975a7286f5fda: Status 404 returned error can't find the container with id 8f7021ffdf799acd41c02645cce626a2214746439f1885b05fb975a7286f5fda Jul 09 22:43:51.545101 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.545065 2568 generic.go:356] "Generic (PLEG): container finished" podID="f856756d-c7e6-4102-8598-cdcf4cfb62fe" containerID="b55b4f2d390af962b390a5fab8c5ce03aa69361d2655083bad70324e76696c9d" exitCode=0 Jul 09 22:43:51.545188 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.545146 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb" event={"ID":"f856756d-c7e6-4102-8598-cdcf4cfb62fe","Type":"ContainerDied","Data":"b55b4f2d390af962b390a5fab8c5ce03aa69361d2655083bad70324e76696c9d"} Jul 09 22:43:51.545283 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.545241 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb" event={"ID":"f856756d-c7e6-4102-8598-cdcf4cfb62fe","Type":"ContainerStarted","Data":"26df126ee6f300c7c3833fce43145ead0386fed91ea105ae1502841f3aab5d92"} Jul 09 22:43:51.886637 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.886602 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4"] Jul 09 22:43:51.890036 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.890017 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4" Jul 09 22:43:51.903632 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:51.903557 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4"] Jul 09 22:43:52.002554 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.002516 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4\" (UID: \"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4" Jul 09 22:43:52.002717 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.002572 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4\" (UID: \"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4" Jul 09 22:43:52.002717 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.002641 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvxdw\" (UniqueName: \"kubernetes.io/projected/3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7-kube-api-access-dvxdw\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4\" (UID: \"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4" Jul 09 22:43:52.103181 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.103147 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4\" (UID: \"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4" Jul 09 22:43:52.103389 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.103195 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvxdw\" (UniqueName: \"kubernetes.io/projected/3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7-kube-api-access-dvxdw\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4\" (UID: \"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4" Jul 09 22:43:52.103389 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.103281 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4\" (UID: \"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4" Jul 09 22:43:52.103601 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.103582 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4\" (UID: \"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4" Jul 09 22:43:52.103682 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.103632 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4\" (UID: \"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4" Jul 09 22:43:52.115916 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.115890 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvxdw\" (UniqueName: \"kubernetes.io/projected/3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7-kube-api-access-dvxdw\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4\" (UID: \"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4" Jul 09 22:43:52.198857 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.198785 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4" Jul 09 22:43:52.290109 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.289766 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79"] Jul 09 22:43:52.296660 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.296635 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79" Jul 09 22:43:52.310824 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.310789 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79"] Jul 09 22:43:52.347257 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.347228 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4"] Jul 09 22:43:52.350204 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:43:52.350164 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3189d1d5_f2db_4cf3_9ef2_f098a0e6a9d7.slice/crio-a1ff47f05514773d2b6568c163868bf77764e8b08fdbb814fbb11a208dda8c87 WatchSource:0}: Error finding container a1ff47f05514773d2b6568c163868bf77764e8b08fdbb814fbb11a208dda8c87: Status 404 returned error can't find the container with id a1ff47f05514773d2b6568c163868bf77764e8b08fdbb814fbb11a208dda8c87 Jul 09 22:43:52.407855 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.407815 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c0737b8-848f-4df6-b244-8cf6cd35cb35-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79\" (UID: \"3c0737b8-848f-4df6-b244-8cf6cd35cb35\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79" Jul 09 22:43:52.408009 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.407896 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c0737b8-848f-4df6-b244-8cf6cd35cb35-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79\" (UID: \"3c0737b8-848f-4df6-b244-8cf6cd35cb35\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79" Jul 09 22:43:52.408009 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.407923 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmxdt\" (UniqueName: \"kubernetes.io/projected/3c0737b8-848f-4df6-b244-8cf6cd35cb35-kube-api-access-xmxdt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79\" (UID: \"3c0737b8-848f-4df6-b244-8cf6cd35cb35\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79" Jul 09 22:43:52.508616 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.508578 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c0737b8-848f-4df6-b244-8cf6cd35cb35-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79\" (UID: \"3c0737b8-848f-4df6-b244-8cf6cd35cb35\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79" Jul 09 22:43:52.508616 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.508620 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmxdt\" (UniqueName: \"kubernetes.io/projected/3c0737b8-848f-4df6-b244-8cf6cd35cb35-kube-api-access-xmxdt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79\" (UID: \"3c0737b8-848f-4df6-b244-8cf6cd35cb35\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79" Jul 09 22:43:52.508835 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.508748 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c0737b8-848f-4df6-b244-8cf6cd35cb35-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79\" (UID: \"3c0737b8-848f-4df6-b244-8cf6cd35cb35\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79" Jul 09 22:43:52.509053 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.509028 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c0737b8-848f-4df6-b244-8cf6cd35cb35-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79\" (UID: \"3c0737b8-848f-4df6-b244-8cf6cd35cb35\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79" Jul 09 22:43:52.509103 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.509088 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c0737b8-848f-4df6-b244-8cf6cd35cb35-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79\" (UID: \"3c0737b8-848f-4df6-b244-8cf6cd35cb35\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79" Jul 09 22:43:52.517823 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.517799 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmxdt\" (UniqueName: \"kubernetes.io/projected/3c0737b8-848f-4df6-b244-8cf6cd35cb35-kube-api-access-xmxdt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79\" (UID: \"3c0737b8-848f-4df6-b244-8cf6cd35cb35\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79" Jul 09 22:43:52.552427 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.552370 2568 generic.go:356] "Generic (PLEG): container finished" podID="b696f428-8899-4dfb-91ee-4ecf429ba33a" containerID="1bc143c8cae66cd3539066e59658a7168ff17a84e0b1bb23a4af4256d86b0c90" exitCode=0 Jul 09 22:43:52.552665 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.552423 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k" event={"ID":"b696f428-8899-4dfb-91ee-4ecf429ba33a","Type":"ContainerDied","Data":"1bc143c8cae66cd3539066e59658a7168ff17a84e0b1bb23a4af4256d86b0c90"} Jul 09 22:43:52.552665 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.552507 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k" event={"ID":"b696f428-8899-4dfb-91ee-4ecf429ba33a","Type":"ContainerStarted","Data":"8f7021ffdf799acd41c02645cce626a2214746439f1885b05fb975a7286f5fda"} Jul 09 22:43:52.554794 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.554756 2568 generic.go:356] "Generic (PLEG): container finished" podID="f856756d-c7e6-4102-8598-cdcf4cfb62fe" containerID="c71602d7f5967343779f01a068d4f404ef94fee6867fd51d0d6de7963fa6bff9" exitCode=0 Jul 09 22:43:52.554917 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.554836 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb" event={"ID":"f856756d-c7e6-4102-8598-cdcf4cfb62fe","Type":"ContainerDied","Data":"c71602d7f5967343779f01a068d4f404ef94fee6867fd51d0d6de7963fa6bff9"} Jul 09 22:43:52.556477 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.556457 2568 generic.go:356] "Generic (PLEG): container finished" podID="3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7" containerID="cf26ff3c339afa36c3c9e02f3f8ae719618db7de7e6dc16a055323b80cfa6431" exitCode=0 Jul 09 22:43:52.556567 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.556522 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4" event={"ID":"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7","Type":"ContainerDied","Data":"cf26ff3c339afa36c3c9e02f3f8ae719618db7de7e6dc16a055323b80cfa6431"} Jul 09 22:43:52.556610 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.556577 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4" event={"ID":"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7","Type":"ContainerStarted","Data":"a1ff47f05514773d2b6568c163868bf77764e8b08fdbb814fbb11a208dda8c87"} Jul 09 22:43:52.618205 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.618184 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79" Jul 09 22:43:52.760865 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:52.760837 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79"] Jul 09 22:43:52.777476 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:43:52.777441 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0737b8_848f_4df6_b244_8cf6cd35cb35.slice/crio-e257790467e5e8dd6fc663f87a6ff0c2a11485d66617facc9778701be88bfbc6 WatchSource:0}: Error finding container e257790467e5e8dd6fc663f87a6ff0c2a11485d66617facc9778701be88bfbc6: Status 404 returned error can't find the container with id e257790467e5e8dd6fc663f87a6ff0c2a11485d66617facc9778701be88bfbc6 Jul 09 22:43:53.565147 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:53.565114 2568 generic.go:356] "Generic (PLEG): container finished" podID="3c0737b8-848f-4df6-b244-8cf6cd35cb35" containerID="2ab72ddcc541fb5a06c9d1a27b327ef8105edd3dddabd9b1548ab08c42468c06" exitCode=0 Jul 09 22:43:53.565582 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:53.565218 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79" event={"ID":"3c0737b8-848f-4df6-b244-8cf6cd35cb35","Type":"ContainerDied","Data":"2ab72ddcc541fb5a06c9d1a27b327ef8105edd3dddabd9b1548ab08c42468c06"} Jul 09 22:43:53.565582 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:53.565274 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79" event={"ID":"3c0737b8-848f-4df6-b244-8cf6cd35cb35","Type":"ContainerStarted","Data":"e257790467e5e8dd6fc663f87a6ff0c2a11485d66617facc9778701be88bfbc6"} Jul 09 22:43:53.567146 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:53.567120 2568 generic.go:356] "Generic (PLEG): container finished" podID="b696f428-8899-4dfb-91ee-4ecf429ba33a" containerID="d61baa25773d3dca33597756f2f0a3785863834546d1d7ea3c5dddfd405d8516" exitCode=0 Jul 09 22:43:53.567231 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:53.567204 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k" event={"ID":"b696f428-8899-4dfb-91ee-4ecf429ba33a","Type":"ContainerDied","Data":"d61baa25773d3dca33597756f2f0a3785863834546d1d7ea3c5dddfd405d8516"} Jul 09 22:43:53.569480 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:53.569453 2568 generic.go:356] "Generic (PLEG): container finished" podID="f856756d-c7e6-4102-8598-cdcf4cfb62fe" containerID="74940c22d1ac7b1bfc7ad0ba27e93cbf794a0a67db7145dffc23d19ebed1c0bd" exitCode=0 Jul 09 22:43:53.569590 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:53.569524 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb" event={"ID":"f856756d-c7e6-4102-8598-cdcf4cfb62fe","Type":"ContainerDied","Data":"74940c22d1ac7b1bfc7ad0ba27e93cbf794a0a67db7145dffc23d19ebed1c0bd"} Jul 09 22:43:53.571374 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:53.571348 2568 generic.go:356] "Generic (PLEG): container finished" podID="3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7" containerID="12efb6011305edb39665b791fcb667357f138a3fd51caa89c69a88eb7ee0b731" exitCode=0 Jul 09 22:43:53.571457 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:53.571413 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4" event={"ID":"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7","Type":"ContainerDied","Data":"12efb6011305edb39665b791fcb667357f138a3fd51caa89c69a88eb7ee0b731"} Jul 09 22:43:54.579565 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:54.579528 2568 generic.go:356] "Generic (PLEG): container finished" podID="b696f428-8899-4dfb-91ee-4ecf429ba33a" containerID="78a8f4105c354a5e7cf5b7308009a77e774128810991234dcb29a1e0b95a4640" exitCode=0 Jul 09 22:43:54.579977 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:54.579608 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k" event={"ID":"b696f428-8899-4dfb-91ee-4ecf429ba33a","Type":"ContainerDied","Data":"78a8f4105c354a5e7cf5b7308009a77e774128810991234dcb29a1e0b95a4640"} Jul 09 22:43:54.581679 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:54.581655 2568 generic.go:356] "Generic (PLEG): container finished" podID="3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7" containerID="12cb473a27863335d39c8e99f0c56fdcffd2d067303c5bcfc1aafd69c1d56e7a" exitCode=0 Jul 09 22:43:54.581785 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:54.581738 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4" event={"ID":"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7","Type":"ContainerDied","Data":"12cb473a27863335d39c8e99f0c56fdcffd2d067303c5bcfc1aafd69c1d56e7a"} Jul 09 22:43:54.583247 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:54.583216 2568 generic.go:356] "Generic (PLEG): container finished" podID="3c0737b8-848f-4df6-b244-8cf6cd35cb35" containerID="6ae43d92e8979bb77661f39047dcdaddcf650db40bdbdae7bf0f7b6496c5af63" exitCode=0 Jul 09 22:43:54.583344 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:54.583295 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79" event={"ID":"3c0737b8-848f-4df6-b244-8cf6cd35cb35","Type":"ContainerDied","Data":"6ae43d92e8979bb77661f39047dcdaddcf650db40bdbdae7bf0f7b6496c5af63"} Jul 09 22:43:54.722400 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:54.722370 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb" Jul 09 22:43:54.827788 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:54.827753 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f856756d-c7e6-4102-8598-cdcf4cfb62fe-util\") pod \"f856756d-c7e6-4102-8598-cdcf4cfb62fe\" (UID: \"f856756d-c7e6-4102-8598-cdcf4cfb62fe\") " Jul 09 22:43:54.827952 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:54.827820 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbdnq\" (UniqueName: \"kubernetes.io/projected/f856756d-c7e6-4102-8598-cdcf4cfb62fe-kube-api-access-sbdnq\") pod \"f856756d-c7e6-4102-8598-cdcf4cfb62fe\" (UID: \"f856756d-c7e6-4102-8598-cdcf4cfb62fe\") " Jul 09 22:43:54.827952 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:54.827851 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f856756d-c7e6-4102-8598-cdcf4cfb62fe-bundle\") pod \"f856756d-c7e6-4102-8598-cdcf4cfb62fe\" (UID: \"f856756d-c7e6-4102-8598-cdcf4cfb62fe\") " Jul 09 22:43:54.828461 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:54.828433 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f856756d-c7e6-4102-8598-cdcf4cfb62fe-bundle" (OuterVolumeSpecName: "bundle") pod "f856756d-c7e6-4102-8598-cdcf4cfb62fe" (UID: "f856756d-c7e6-4102-8598-cdcf4cfb62fe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:43:54.830002 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:54.829973 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f856756d-c7e6-4102-8598-cdcf4cfb62fe-kube-api-access-sbdnq" (OuterVolumeSpecName: "kube-api-access-sbdnq") pod "f856756d-c7e6-4102-8598-cdcf4cfb62fe" (UID: "f856756d-c7e6-4102-8598-cdcf4cfb62fe"). InnerVolumeSpecName "kube-api-access-sbdnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:43:54.833022 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:54.833001 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f856756d-c7e6-4102-8598-cdcf4cfb62fe-util" (OuterVolumeSpecName: "util") pod "f856756d-c7e6-4102-8598-cdcf4cfb62fe" (UID: "f856756d-c7e6-4102-8598-cdcf4cfb62fe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:43:54.928623 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:54.928573 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f856756d-c7e6-4102-8598-cdcf4cfb62fe-util\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:43:54.928623 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:54.928613 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbdnq\" (UniqueName: \"kubernetes.io/projected/f856756d-c7e6-4102-8598-cdcf4cfb62fe-kube-api-access-sbdnq\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:43:54.928623 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:54.928627 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f856756d-c7e6-4102-8598-cdcf4cfb62fe-bundle\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:43:55.590680 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.590647 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb" Jul 09 22:43:55.590680 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.590671 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb" event={"ID":"f856756d-c7e6-4102-8598-cdcf4cfb62fe","Type":"ContainerDied","Data":"26df126ee6f300c7c3833fce43145ead0386fed91ea105ae1502841f3aab5d92"} Jul 09 22:43:55.591262 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.590719 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26df126ee6f300c7c3833fce43145ead0386fed91ea105ae1502841f3aab5d92" Jul 09 22:43:55.592866 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.592837 2568 generic.go:356] "Generic (PLEG): container finished" podID="3c0737b8-848f-4df6-b244-8cf6cd35cb35" containerID="09f796b437044b91bdfbf22430aa44be6b22309baf7a6f4a97d208cba9d9884f" exitCode=0 Jul 09 22:43:55.593038 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.592921 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79" event={"ID":"3c0737b8-848f-4df6-b244-8cf6cd35cb35","Type":"ContainerDied","Data":"09f796b437044b91bdfbf22430aa44be6b22309baf7a6f4a97d208cba9d9884f"} Jul 09 22:43:55.732709 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.732671 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4" Jul 09 22:43:55.772891 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.772862 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k" Jul 09 22:43:55.837121 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.837034 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b696f428-8899-4dfb-91ee-4ecf429ba33a-util\") pod \"b696f428-8899-4dfb-91ee-4ecf429ba33a\" (UID: \"b696f428-8899-4dfb-91ee-4ecf429ba33a\") " Jul 09 22:43:55.837121 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.837084 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvxdw\" (UniqueName: \"kubernetes.io/projected/3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7-kube-api-access-dvxdw\") pod \"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7\" (UID: \"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7\") " Jul 09 22:43:55.837398 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.837144 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b696f428-8899-4dfb-91ee-4ecf429ba33a-bundle\") pod \"b696f428-8899-4dfb-91ee-4ecf429ba33a\" (UID: \"b696f428-8899-4dfb-91ee-4ecf429ba33a\") " Jul 09 22:43:55.837398 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.837178 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5v4d\" (UniqueName: \"kubernetes.io/projected/b696f428-8899-4dfb-91ee-4ecf429ba33a-kube-api-access-p5v4d\") pod \"b696f428-8899-4dfb-91ee-4ecf429ba33a\" (UID: \"b696f428-8899-4dfb-91ee-4ecf429ba33a\") " Jul 09 22:43:55.837398 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.837238 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7-util\") pod \"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7\" (UID: \"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7\") " Jul 09 22:43:55.837398 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.837299 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7-bundle\") pod \"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7\" (UID: \"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7\") " Jul 09 22:43:55.837849 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.837812 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7-bundle" (OuterVolumeSpecName: "bundle") pod "3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7" (UID: "3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:43:55.837970 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.837854 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b696f428-8899-4dfb-91ee-4ecf429ba33a-bundle" (OuterVolumeSpecName: "bundle") pod "b696f428-8899-4dfb-91ee-4ecf429ba33a" (UID: "b696f428-8899-4dfb-91ee-4ecf429ba33a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:43:55.839404 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.839381 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7-kube-api-access-dvxdw" (OuterVolumeSpecName: "kube-api-access-dvxdw") pod "3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7" (UID: "3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7"). InnerVolumeSpecName "kube-api-access-dvxdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:43:55.839652 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.839637 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b696f428-8899-4dfb-91ee-4ecf429ba33a-kube-api-access-p5v4d" (OuterVolumeSpecName: "kube-api-access-p5v4d") pod "b696f428-8899-4dfb-91ee-4ecf429ba33a" (UID: "b696f428-8899-4dfb-91ee-4ecf429ba33a"). InnerVolumeSpecName "kube-api-access-p5v4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:43:55.843145 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.843111 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7-util" (OuterVolumeSpecName: "util") pod "3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7" (UID: "3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:43:55.843397 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.843378 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b696f428-8899-4dfb-91ee-4ecf429ba33a-util" (OuterVolumeSpecName: "util") pod "b696f428-8899-4dfb-91ee-4ecf429ba33a" (UID: "b696f428-8899-4dfb-91ee-4ecf429ba33a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:43:55.938774 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.938724 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvxdw\" (UniqueName: \"kubernetes.io/projected/3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7-kube-api-access-dvxdw\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:43:55.938774 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.938765 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b696f428-8899-4dfb-91ee-4ecf429ba33a-bundle\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:43:55.938774 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.938775 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5v4d\" (UniqueName: \"kubernetes.io/projected/b696f428-8899-4dfb-91ee-4ecf429ba33a-kube-api-access-p5v4d\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:43:55.938774 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.938785 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7-util\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:43:55.939042 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.938795 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7-bundle\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:43:55.939042 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:55.938803 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b696f428-8899-4dfb-91ee-4ecf429ba33a-util\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:43:56.599705 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:56.599676 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k" Jul 09 22:43:56.600145 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:56.599676 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k" event={"ID":"b696f428-8899-4dfb-91ee-4ecf429ba33a","Type":"ContainerDied","Data":"8f7021ffdf799acd41c02645cce626a2214746439f1885b05fb975a7286f5fda"} Jul 09 22:43:56.600145 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:56.599801 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f7021ffdf799acd41c02645cce626a2214746439f1885b05fb975a7286f5fda" Jul 09 22:43:56.601698 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:56.601674 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4" Jul 09 22:43:56.601824 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:56.601758 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4" event={"ID":"3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7","Type":"ContainerDied","Data":"a1ff47f05514773d2b6568c163868bf77764e8b08fdbb814fbb11a208dda8c87"} Jul 09 22:43:56.601824 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:56.601802 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1ff47f05514773d2b6568c163868bf77764e8b08fdbb814fbb11a208dda8c87" Jul 09 22:43:56.739715 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:56.739682 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79" Jul 09 22:43:56.847150 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:56.847055 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmxdt\" (UniqueName: \"kubernetes.io/projected/3c0737b8-848f-4df6-b244-8cf6cd35cb35-kube-api-access-xmxdt\") pod \"3c0737b8-848f-4df6-b244-8cf6cd35cb35\" (UID: \"3c0737b8-848f-4df6-b244-8cf6cd35cb35\") " Jul 09 22:43:56.847327 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:56.847151 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c0737b8-848f-4df6-b244-8cf6cd35cb35-bundle\") pod \"3c0737b8-848f-4df6-b244-8cf6cd35cb35\" (UID: \"3c0737b8-848f-4df6-b244-8cf6cd35cb35\") " Jul 09 22:43:56.847327 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:56.847197 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c0737b8-848f-4df6-b244-8cf6cd35cb35-util\") pod \"3c0737b8-848f-4df6-b244-8cf6cd35cb35\" (UID: \"3c0737b8-848f-4df6-b244-8cf6cd35cb35\") " Jul 09 22:43:56.847699 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:56.847672 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0737b8-848f-4df6-b244-8cf6cd35cb35-bundle" (OuterVolumeSpecName: "bundle") pod "3c0737b8-848f-4df6-b244-8cf6cd35cb35" (UID: "3c0737b8-848f-4df6-b244-8cf6cd35cb35"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:43:56.849171 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:56.849142 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0737b8-848f-4df6-b244-8cf6cd35cb35-kube-api-access-xmxdt" (OuterVolumeSpecName: "kube-api-access-xmxdt") pod "3c0737b8-848f-4df6-b244-8cf6cd35cb35" (UID: "3c0737b8-848f-4df6-b244-8cf6cd35cb35"). InnerVolumeSpecName "kube-api-access-xmxdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:43:56.852460 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:56.852438 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0737b8-848f-4df6-b244-8cf6cd35cb35-util" (OuterVolumeSpecName: "util") pod "3c0737b8-848f-4df6-b244-8cf6cd35cb35" (UID: "3c0737b8-848f-4df6-b244-8cf6cd35cb35"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:43:56.948779 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:56.948729 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c0737b8-848f-4df6-b244-8cf6cd35cb35-bundle\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:43:56.948779 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:56.948773 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c0737b8-848f-4df6-b244-8cf6cd35cb35-util\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:43:56.948779 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:56.948782 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmxdt\" (UniqueName: \"kubernetes.io/projected/3c0737b8-848f-4df6-b244-8cf6cd35cb35-kube-api-access-xmxdt\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:43:57.613240 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:57.613180 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79" event={"ID":"3c0737b8-848f-4df6-b244-8cf6cd35cb35","Type":"ContainerDied","Data":"e257790467e5e8dd6fc663f87a6ff0c2a11485d66617facc9778701be88bfbc6"} Jul 09 22:43:57.613654 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:57.613277 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e257790467e5e8dd6fc663f87a6ff0c2a11485d66617facc9778701be88bfbc6" Jul 09 22:43:57.613654 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:43:57.613182 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79" Jul 09 22:44:04.086014 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:04.085979 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:08.191743 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:08.191703 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:10.093534 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:10.093476 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:10.679053 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:10.678986 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:10.849008 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:10.848973 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:10.955897 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:10.955802 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:14.286535 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:14.286480 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:17.293702 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:17.293660 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:19.857444 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:19.857412 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:20.133623 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:20.133532 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:21.823806 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:21.823766 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:21.923444 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:21.923414 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:22.086654 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:22.086565 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:22.188063 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:22.188023 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:31.551635 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:31.551599 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:31.658041 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:31.658006 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:32.295883 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:32.295851 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:32.395568 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:32.395531 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:32.890070 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:32.890037 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:32.988060 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:32.988021 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:33.124223 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:33.124189 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:35.514014 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:35.513979 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:36.880917 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:36.880880 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:36.977064 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:36.977031 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:38.685418 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:38.685377 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:38.782834 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:38.782799 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:45.687835 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:45.687795 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:45.754401 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:45.754356 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:50.714540 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:50.714411 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:54.313025 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:54.312991 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:54.415259 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:54.415223 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:57.356181 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:57.356147 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:57.485260 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:57.485213 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:59.783021 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:59.782979 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:44:59.880739 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:44:59.880705 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:04.672368 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:04.672330 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:05.352245 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:05.352205 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:05.529263 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:05.529224 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:05.611225 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:05.611128 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:10.956365 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:10.956331 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:11.079095 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:11.079060 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:11.183199 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:11.183162 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:11.823281 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:11.823250 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:11.894558 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:11.894517 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:15.733410 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:15.733362 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:15.815821 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:15.815783 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:15.916042 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:15.916006 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:16.012070 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:16.011960 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:17.741581 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:17.741549 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:17.850671 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:17.850633 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:20.187364 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:20.187325 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:20.287627 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:20.287587 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:32.353547 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:32.353513 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:32.455989 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:32.455952 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:32.997481 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:32.997443 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:45.407747 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:45.407711 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:45.762888 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:45.762794 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:53.140968 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.140931 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:53.149768 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.149738 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-76d6bc8bd-hrtkm"] Jul 09 22:45:53.150051 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150039 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7" containerName="extract" Jul 09 22:45:53.150096 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150053 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7" containerName="extract" Jul 09 22:45:53.150096 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150064 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7" containerName="pull" Jul 09 22:45:53.150096 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150069 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7" containerName="pull" Jul 09 22:45:53.150096 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150075 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c0737b8-848f-4df6-b244-8cf6cd35cb35" containerName="pull" Jul 09 22:45:53.150096 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150080 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c0737b8-848f-4df6-b244-8cf6cd35cb35" containerName="pull" Jul 09 22:45:53.150096 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150088 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b696f428-8899-4dfb-91ee-4ecf429ba33a" containerName="util" Jul 09 22:45:53.150096 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150094 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b696f428-8899-4dfb-91ee-4ecf429ba33a" containerName="util" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150100 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c0737b8-848f-4df6-b244-8cf6cd35cb35" containerName="util" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150105 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c0737b8-848f-4df6-b244-8cf6cd35cb35" containerName="util" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150110 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b696f428-8899-4dfb-91ee-4ecf429ba33a" containerName="pull" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150115 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b696f428-8899-4dfb-91ee-4ecf429ba33a" containerName="pull" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150123 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7" containerName="util" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150127 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7" containerName="util" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150134 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c0737b8-848f-4df6-b244-8cf6cd35cb35" containerName="extract" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150139 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c0737b8-848f-4df6-b244-8cf6cd35cb35" containerName="extract" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150145 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f856756d-c7e6-4102-8598-cdcf4cfb62fe" containerName="util" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150150 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f856756d-c7e6-4102-8598-cdcf4cfb62fe" containerName="util" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150157 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f856756d-c7e6-4102-8598-cdcf4cfb62fe" containerName="extract" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150162 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f856756d-c7e6-4102-8598-cdcf4cfb62fe" containerName="extract" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150167 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b696f428-8899-4dfb-91ee-4ecf429ba33a" containerName="extract" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150172 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b696f428-8899-4dfb-91ee-4ecf429ba33a" containerName="extract" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150184 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f856756d-c7e6-4102-8598-cdcf4cfb62fe" containerName="pull" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150188 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f856756d-c7e6-4102-8598-cdcf4cfb62fe" containerName="pull" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150237 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c0737b8-848f-4df6-b244-8cf6cd35cb35" containerName="extract" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150244 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7" containerName="extract" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150252 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b696f428-8899-4dfb-91ee-4ecf429ba33a" containerName="extract" Jul 09 22:45:53.150299 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.150257 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f856756d-c7e6-4102-8598-cdcf4cfb62fe" containerName="extract" Jul 09 22:45:53.153107 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.153091 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76d6bc8bd-hrtkm" Jul 09 22:45:53.156367 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.156338 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wdzbn\"" Jul 09 22:45:53.156534 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.156466 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 22:45:53.157622 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.157604 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 22:45:53.172327 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.172284 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76d6bc8bd-hrtkm"] Jul 09 22:45:53.249423 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.249375 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:53.260461 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.260426 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j7hz\" (UniqueName: \"kubernetes.io/projected/e081f0be-9233-4cde-bf76-41517a2e3f2e-kube-api-access-8j7hz\") pod \"maas-controller-76d6bc8bd-hrtkm\" (UID: \"e081f0be-9233-4cde-bf76-41517a2e3f2e\") " pod="opendatahub/maas-controller-76d6bc8bd-hrtkm" Jul 09 22:45:53.260646 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.260468 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e081f0be-9233-4cde-bf76-41517a2e3f2e-webhook-cert\") pod \"maas-controller-76d6bc8bd-hrtkm\" (UID: \"e081f0be-9233-4cde-bf76-41517a2e3f2e\") " pod="opendatahub/maas-controller-76d6bc8bd-hrtkm" Jul 09 22:45:53.350756 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.350722 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:53.360914 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.360883 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j7hz\" (UniqueName: \"kubernetes.io/projected/e081f0be-9233-4cde-bf76-41517a2e3f2e-kube-api-access-8j7hz\") pod \"maas-controller-76d6bc8bd-hrtkm\" (UID: \"e081f0be-9233-4cde-bf76-41517a2e3f2e\") " pod="opendatahub/maas-controller-76d6bc8bd-hrtkm" Jul 09 22:45:53.361048 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.360925 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e081f0be-9233-4cde-bf76-41517a2e3f2e-webhook-cert\") pod \"maas-controller-76d6bc8bd-hrtkm\" (UID: \"e081f0be-9233-4cde-bf76-41517a2e3f2e\") " pod="opendatahub/maas-controller-76d6bc8bd-hrtkm" Jul 09 22:45:53.361119 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:45:53.361098 2568 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 09 22:45:53.361192 ip-10-0-135-80 kubenswrapper[2568]: E0709 22:45:53.361181 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e081f0be-9233-4cde-bf76-41517a2e3f2e-webhook-cert podName:e081f0be-9233-4cde-bf76-41517a2e3f2e nodeName:}" failed. No retries permitted until 2026-07-09 22:45:53.861155214 +0000 UTC m=+778.402233987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/e081f0be-9233-4cde-bf76-41517a2e3f2e-webhook-cert") pod "maas-controller-76d6bc8bd-hrtkm" (UID: "e081f0be-9233-4cde-bf76-41517a2e3f2e") : secret "maas-controller-webhook-cert" not found Jul 09 22:45:53.370901 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.370862 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j7hz\" (UniqueName: \"kubernetes.io/projected/e081f0be-9233-4cde-bf76-41517a2e3f2e-kube-api-access-8j7hz\") pod \"maas-controller-76d6bc8bd-hrtkm\" (UID: \"e081f0be-9233-4cde-bf76-41517a2e3f2e\") " pod="opendatahub/maas-controller-76d6bc8bd-hrtkm" Jul 09 22:45:53.463784 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.463697 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:53.865948 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.865914 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e081f0be-9233-4cde-bf76-41517a2e3f2e-webhook-cert\") pod \"maas-controller-76d6bc8bd-hrtkm\" (UID: \"e081f0be-9233-4cde-bf76-41517a2e3f2e\") " pod="opendatahub/maas-controller-76d6bc8bd-hrtkm" Jul 09 22:45:53.868528 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:53.868471 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e081f0be-9233-4cde-bf76-41517a2e3f2e-webhook-cert\") pod \"maas-controller-76d6bc8bd-hrtkm\" (UID: \"e081f0be-9233-4cde-bf76-41517a2e3f2e\") " pod="opendatahub/maas-controller-76d6bc8bd-hrtkm" Jul 09 22:45:54.063255 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:54.063220 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76d6bc8bd-hrtkm" Jul 09 22:45:54.205577 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:54.205554 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76d6bc8bd-hrtkm"] Jul 09 22:45:54.207710 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:45:54.207683 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode081f0be_9233_4cde_bf76_41517a2e3f2e.slice/crio-4701562838a0347c99351d656351b508a10b276abd6c6e1ad11ffb237ba241f9 WatchSource:0}: Error finding container 4701562838a0347c99351d656351b508a10b276abd6c6e1ad11ffb237ba241f9: Status 404 returned error can't find the container with id 4701562838a0347c99351d656351b508a10b276abd6c6e1ad11ffb237ba241f9 Jul 09 22:45:54.208933 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:54.208914 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 22:45:55.213783 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:55.213698 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76d6bc8bd-hrtkm" event={"ID":"e081f0be-9233-4cde-bf76-41517a2e3f2e","Type":"ContainerStarted","Data":"4701562838a0347c99351d656351b508a10b276abd6c6e1ad11ffb237ba241f9"} Jul 09 22:45:56.050031 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:56.050006 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 22:45:56.208246 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:56.208157 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:45:56.221247 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:56.221151 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76d6bc8bd-hrtkm" event={"ID":"e081f0be-9233-4cde-bf76-41517a2e3f2e","Type":"ContainerStarted","Data":"4d6c1f49cfb9b962c9906478e529e2879e3e765bc631d8ab0ec0fa3768db77dc"} Jul 09 22:45:56.221766 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:56.221385 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-76d6bc8bd-hrtkm" Jul 09 22:45:56.253231 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:56.253175 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-76d6bc8bd-hrtkm" podStartSLOduration=1.415133732 podStartE2EDuration="3.253150506s" totalImagesPullingTime="1.838016774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:45:56.250392319 +0000 UTC m=+780.791471113" watchObservedRunningTime="2026-07-09 22:45:56.253150506 +0000 UTC m=+780.794229300" Jul 09 22:45:56.308400 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:45:56.308362 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:46:00.455322 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:00.455280 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:46:07.119431 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:07.119388 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76d6bc8bd-hrtkm"] Jul 09 22:46:07.119900 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:07.119609 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-76d6bc8bd-hrtkm" podUID="e081f0be-9233-4cde-bf76-41517a2e3f2e" containerName="manager" containerID="cri-o://4d6c1f49cfb9b962c9906478e529e2879e3e765bc631d8ab0ec0fa3768db77dc" gracePeriod=30 Jul 09 22:46:07.128059 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:07.128026 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-76d6bc8bd-hrtkm" Jul 09 22:46:07.280888 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:07.280854 2568 generic.go:356] "Generic (PLEG): container finished" podID="e081f0be-9233-4cde-bf76-41517a2e3f2e" containerID="4d6c1f49cfb9b962c9906478e529e2879e3e765bc631d8ab0ec0fa3768db77dc" exitCode=0 Jul 09 22:46:07.280888 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:07.280898 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76d6bc8bd-hrtkm" event={"ID":"e081f0be-9233-4cde-bf76-41517a2e3f2e","Type":"ContainerDied","Data":"4d6c1f49cfb9b962c9906478e529e2879e3e765bc631d8ab0ec0fa3768db77dc"} Jul 09 22:46:07.378012 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:07.377926 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76d6bc8bd-hrtkm" Jul 09 22:46:07.479379 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:07.479338 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j7hz\" (UniqueName: \"kubernetes.io/projected/e081f0be-9233-4cde-bf76-41517a2e3f2e-kube-api-access-8j7hz\") pod \"e081f0be-9233-4cde-bf76-41517a2e3f2e\" (UID: \"e081f0be-9233-4cde-bf76-41517a2e3f2e\") " Jul 09 22:46:07.479626 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:07.479396 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e081f0be-9233-4cde-bf76-41517a2e3f2e-webhook-cert\") pod \"e081f0be-9233-4cde-bf76-41517a2e3f2e\" (UID: \"e081f0be-9233-4cde-bf76-41517a2e3f2e\") " Jul 09 22:46:07.481780 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:07.481738 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e081f0be-9233-4cde-bf76-41517a2e3f2e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e081f0be-9233-4cde-bf76-41517a2e3f2e" (UID: "e081f0be-9233-4cde-bf76-41517a2e3f2e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:46:07.481780 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:07.481764 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e081f0be-9233-4cde-bf76-41517a2e3f2e-kube-api-access-8j7hz" (OuterVolumeSpecName: "kube-api-access-8j7hz") pod "e081f0be-9233-4cde-bf76-41517a2e3f2e" (UID: "e081f0be-9233-4cde-bf76-41517a2e3f2e"). InnerVolumeSpecName "kube-api-access-8j7hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:46:07.580418 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:07.580364 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j7hz\" (UniqueName: \"kubernetes.io/projected/e081f0be-9233-4cde-bf76-41517a2e3f2e-kube-api-access-8j7hz\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:46:07.580418 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:07.580405 2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e081f0be-9233-4cde-bf76-41517a2e3f2e-webhook-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 22:46:08.287520 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:08.287454 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76d6bc8bd-hrtkm" event={"ID":"e081f0be-9233-4cde-bf76-41517a2e3f2e","Type":"ContainerDied","Data":"4701562838a0347c99351d656351b508a10b276abd6c6e1ad11ffb237ba241f9"} Jul 09 22:46:08.287520 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:08.287513 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76d6bc8bd-hrtkm" Jul 09 22:46:08.288105 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:08.287545 2568 scope.go:117] "RemoveContainer" containerID="4d6c1f49cfb9b962c9906478e529e2879e3e765bc631d8ab0ec0fa3768db77dc" Jul 09 22:46:08.306542 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:08.306437 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76d6bc8bd-hrtkm"] Jul 09 22:46:08.308512 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:08.308455 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-76d6bc8bd-hrtkm"] Jul 09 22:46:10.051600 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:10.051561 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e081f0be-9233-4cde-bf76-41517a2e3f2e" path="/var/lib/kubelet/pods/e081f0be-9233-4cde-bf76-41517a2e3f2e/volumes" Jul 09 22:46:27.671234 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:27.671139 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:46:28.327120 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.327089 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:46:28.390533 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.390500 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b54d47c47-crcpg"] Jul 09 22:46:28.390824 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.390811 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e081f0be-9233-4cde-bf76-41517a2e3f2e" containerName="manager" Jul 09 22:46:28.390882 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.390825 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e081f0be-9233-4cde-bf76-41517a2e3f2e" containerName="manager" Jul 09 22:46:28.390919 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.390906 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e081f0be-9233-4cde-bf76-41517a2e3f2e" containerName="manager" Jul 09 22:46:28.393845 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.393824 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b54d47c47-crcpg" Jul 09 22:46:28.397355 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.397332 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 09 22:46:28.398228 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.398200 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 22:46:28.398353 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.398266 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c2r6j\"" Jul 09 22:46:28.398353 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.398196 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 22:46:28.409865 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.409831 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b54d47c47-crcpg"] Jul 09 22:46:28.466239 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.466197 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51fa8f1c-8729-42f5-b123-12fc4d7909a9-maas-api-tls\") pod \"maas-api-7b54d47c47-crcpg\" (UID: \"51fa8f1c-8729-42f5-b123-12fc4d7909a9\") " pod="odh-ai-gateway-infra/maas-api-7b54d47c47-crcpg" Jul 09 22:46:28.466239 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.466234 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6vvc\" (UniqueName: \"kubernetes.io/projected/51fa8f1c-8729-42f5-b123-12fc4d7909a9-kube-api-access-j6vvc\") pod \"maas-api-7b54d47c47-crcpg\" (UID: \"51fa8f1c-8729-42f5-b123-12fc4d7909a9\") " pod="odh-ai-gateway-infra/maas-api-7b54d47c47-crcpg" Jul 09 22:46:28.466702 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.466339 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fa8f1c-8729-42f5-b123-12fc4d7909a9-openshift-service-ca\") pod \"maas-api-7b54d47c47-crcpg\" (UID: \"51fa8f1c-8729-42f5-b123-12fc4d7909a9\") " pod="odh-ai-gateway-infra/maas-api-7b54d47c47-crcpg" Jul 09 22:46:28.472423 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.472382 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:46:28.566996 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.566957 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fa8f1c-8729-42f5-b123-12fc4d7909a9-openshift-service-ca\") pod \"maas-api-7b54d47c47-crcpg\" (UID: \"51fa8f1c-8729-42f5-b123-12fc4d7909a9\") " pod="odh-ai-gateway-infra/maas-api-7b54d47c47-crcpg" Jul 09 22:46:28.567213 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.567015 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51fa8f1c-8729-42f5-b123-12fc4d7909a9-maas-api-tls\") pod \"maas-api-7b54d47c47-crcpg\" (UID: \"51fa8f1c-8729-42f5-b123-12fc4d7909a9\") " pod="odh-ai-gateway-infra/maas-api-7b54d47c47-crcpg" Jul 09 22:46:28.567213 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.567129 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6vvc\" (UniqueName: \"kubernetes.io/projected/51fa8f1c-8729-42f5-b123-12fc4d7909a9-kube-api-access-j6vvc\") pod \"maas-api-7b54d47c47-crcpg\" (UID: \"51fa8f1c-8729-42f5-b123-12fc4d7909a9\") " pod="odh-ai-gateway-infra/maas-api-7b54d47c47-crcpg" Jul 09 22:46:28.567774 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.567743 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fa8f1c-8729-42f5-b123-12fc4d7909a9-openshift-service-ca\") pod \"maas-api-7b54d47c47-crcpg\" (UID: \"51fa8f1c-8729-42f5-b123-12fc4d7909a9\") " pod="odh-ai-gateway-infra/maas-api-7b54d47c47-crcpg" Jul 09 22:46:28.569547 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.569524 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51fa8f1c-8729-42f5-b123-12fc4d7909a9-maas-api-tls\") pod \"maas-api-7b54d47c47-crcpg\" (UID: \"51fa8f1c-8729-42f5-b123-12fc4d7909a9\") " pod="odh-ai-gateway-infra/maas-api-7b54d47c47-crcpg" Jul 09 22:46:28.577202 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.577134 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6vvc\" (UniqueName: \"kubernetes.io/projected/51fa8f1c-8729-42f5-b123-12fc4d7909a9-kube-api-access-j6vvc\") pod \"maas-api-7b54d47c47-crcpg\" (UID: \"51fa8f1c-8729-42f5-b123-12fc4d7909a9\") " pod="odh-ai-gateway-infra/maas-api-7b54d47c47-crcpg" Jul 09 22:46:28.704294 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.704256 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b54d47c47-crcpg" Jul 09 22:46:28.865233 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:46:28.865195 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fa8f1c_8729_42f5_b123_12fc4d7909a9.slice/crio-79c683b0575819914d6dbfb2c615c62b23b975cf8769c8a2a933adff42f362fc WatchSource:0}: Error finding container 79c683b0575819914d6dbfb2c615c62b23b975cf8769c8a2a933adff42f362fc: Status 404 returned error can't find the container with id 79c683b0575819914d6dbfb2c615c62b23b975cf8769c8a2a933adff42f362fc Jul 09 22:46:28.865900 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.865871 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b54d47c47-crcpg"] Jul 09 22:46:28.994608 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:28.994569 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:46:29.083001 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:29.082967 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:46:29.397561 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:29.397522 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b54d47c47-crcpg" event={"ID":"51fa8f1c-8729-42f5-b123-12fc4d7909a9","Type":"ContainerStarted","Data":"79c683b0575819914d6dbfb2c615c62b23b975cf8769c8a2a933adff42f362fc"} Jul 09 22:46:31.411295 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:31.411260 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b54d47c47-crcpg" event={"ID":"51fa8f1c-8729-42f5-b123-12fc4d7909a9","Type":"ContainerStarted","Data":"fa2815c4839d22aaa965a1e3b287e55a4816b1fae75fb427869d5c970ec5f9aa"} Jul 09 22:46:31.411761 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:31.411338 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b54d47c47-crcpg" Jul 09 22:46:31.431801 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:31.431749 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b54d47c47-crcpg" podStartSLOduration=1.7471058350000002 podStartE2EDuration="3.431732207s" totalImagesPullingTime="1.684626372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:46:31.430222187 +0000 UTC m=+815.971301006" watchObservedRunningTime="2026-07-09 22:46:31.431732207 +0000 UTC m=+815.972811060" Jul 09 22:46:37.421605 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:37.421576 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b54d47c47-crcpg" Jul 09 22:46:38.791119 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:38.791086 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:46:38.886704 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:38.886665 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:46:47.320213 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:47.320173 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:46:51.720569 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.720532 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw"] Jul 09 22:46:51.723904 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.723884 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:51.728414 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.728382 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-52nhr\"" Jul 09 22:46:51.728611 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.728451 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 09 22:46:51.728611 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.728392 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 22:46:51.728611 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.728397 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 22:46:51.738565 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.738535 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw"] Jul 09 22:46:51.765154 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.765106 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:46:51.870456 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.870423 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee10461-24fb-4eb3-8289-4ab34c9cbe9e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pc2jw\" (UID: \"cee10461-24fb-4eb3-8289-4ab34c9cbe9e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:51.870456 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.870457 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee10461-24fb-4eb3-8289-4ab34c9cbe9e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pc2jw\" (UID: \"cee10461-24fb-4eb3-8289-4ab34c9cbe9e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:51.870737 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.870517 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee10461-24fb-4eb3-8289-4ab34c9cbe9e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pc2jw\" (UID: \"cee10461-24fb-4eb3-8289-4ab34c9cbe9e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:51.870737 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.870592 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f8pq\" (UniqueName: \"kubernetes.io/projected/cee10461-24fb-4eb3-8289-4ab34c9cbe9e-kube-api-access-9f8pq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pc2jw\" (UID: \"cee10461-24fb-4eb3-8289-4ab34c9cbe9e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:51.870737 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.870655 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee10461-24fb-4eb3-8289-4ab34c9cbe9e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pc2jw\" (UID: \"cee10461-24fb-4eb3-8289-4ab34c9cbe9e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:51.870737 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.870722 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee10461-24fb-4eb3-8289-4ab34c9cbe9e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pc2jw\" (UID: \"cee10461-24fb-4eb3-8289-4ab34c9cbe9e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:51.971569 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.971458 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee10461-24fb-4eb3-8289-4ab34c9cbe9e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pc2jw\" (UID: \"cee10461-24fb-4eb3-8289-4ab34c9cbe9e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:51.971569 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.971544 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee10461-24fb-4eb3-8289-4ab34c9cbe9e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pc2jw\" (UID: \"cee10461-24fb-4eb3-8289-4ab34c9cbe9e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:51.971811 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.971577 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee10461-24fb-4eb3-8289-4ab34c9cbe9e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pc2jw\" (UID: \"cee10461-24fb-4eb3-8289-4ab34c9cbe9e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:51.971811 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.971597 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee10461-24fb-4eb3-8289-4ab34c9cbe9e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pc2jw\" (UID: \"cee10461-24fb-4eb3-8289-4ab34c9cbe9e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:51.971811 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.971644 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee10461-24fb-4eb3-8289-4ab34c9cbe9e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pc2jw\" (UID: \"cee10461-24fb-4eb3-8289-4ab34c9cbe9e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:51.971811 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.971673 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f8pq\" (UniqueName: \"kubernetes.io/projected/cee10461-24fb-4eb3-8289-4ab34c9cbe9e-kube-api-access-9f8pq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pc2jw\" (UID: \"cee10461-24fb-4eb3-8289-4ab34c9cbe9e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:51.972006 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.971986 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee10461-24fb-4eb3-8289-4ab34c9cbe9e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pc2jw\" (UID: \"cee10461-24fb-4eb3-8289-4ab34c9cbe9e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:51.972072 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.972016 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee10461-24fb-4eb3-8289-4ab34c9cbe9e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pc2jw\" (UID: \"cee10461-24fb-4eb3-8289-4ab34c9cbe9e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:51.972128 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.972099 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee10461-24fb-4eb3-8289-4ab34c9cbe9e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pc2jw\" (UID: \"cee10461-24fb-4eb3-8289-4ab34c9cbe9e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:51.973835 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.973807 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee10461-24fb-4eb3-8289-4ab34c9cbe9e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pc2jw\" (UID: \"cee10461-24fb-4eb3-8289-4ab34c9cbe9e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:51.974087 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.974068 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee10461-24fb-4eb3-8289-4ab34c9cbe9e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pc2jw\" (UID: \"cee10461-24fb-4eb3-8289-4ab34c9cbe9e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:51.984296 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:51.984274 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f8pq\" (UniqueName: \"kubernetes.io/projected/cee10461-24fb-4eb3-8289-4ab34c9cbe9e-kube-api-access-9f8pq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pc2jw\" (UID: \"cee10461-24fb-4eb3-8289-4ab34c9cbe9e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:52.034200 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:52.034163 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:46:52.170712 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:52.170677 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw"] Jul 09 22:46:52.173811 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:46:52.173785 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee10461_24fb_4eb3_8289_4ab34c9cbe9e.slice/crio-195b12e6714feab67b8a47914d199a8e4e36ae13530d6d9421884f4e2a2ff520 WatchSource:0}: Error finding container 195b12e6714feab67b8a47914d199a8e4e36ae13530d6d9421884f4e2a2ff520: Status 404 returned error can't find the container with id 195b12e6714feab67b8a47914d199a8e4e36ae13530d6d9421884f4e2a2ff520 Jul 09 22:46:52.519472 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:52.519429 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" event={"ID":"cee10461-24fb-4eb3-8289-4ab34c9cbe9e","Type":"ContainerStarted","Data":"195b12e6714feab67b8a47914d199a8e4e36ae13530d6d9421884f4e2a2ff520"} Jul 09 22:46:53.526071 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:53.525985 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" event={"ID":"cee10461-24fb-4eb3-8289-4ab34c9cbe9e","Type":"ContainerStarted","Data":"9f8cef1ce06f5b23975c1014575a71fb796fffdbc34f3850b2e44d3dd1b5ef16"} Jul 09 22:46:53.564727 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:53.564670 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" podStartSLOduration=1.5553133529999998 podStartE2EDuration="2.564652766s" totalImagesPullingTime="1.009339413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:46:53.561200139 +0000 UTC m=+838.102278932" watchObservedRunningTime="2026-07-09 22:46:53.564652766 +0000 UTC m=+838.105731567" Jul 09 22:46:57.603261 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:57.603227 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:46:57.703273 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:46:57.703235 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:47:02.035323 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:02.035280 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:47:02.035857 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:02.035366 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:47:02.048987 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:02.048959 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:47:02.582096 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:02.582068 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pc2jw" Jul 09 22:47:10.651512 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:10.651460 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:47:16.780832 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.780797 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84"] Jul 09 22:47:16.785678 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.785658 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:16.790335 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.790311 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 09 22:47:16.797690 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.797659 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84"] Jul 09 22:47:16.877278 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.877240 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:47:16.879311 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.879278 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6ppt\" (UniqueName: \"kubernetes.io/projected/12282b71-4b77-497a-abac-9042f10d922c-kube-api-access-c6ppt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84\" (UID: \"12282b71-4b77-497a-abac-9042f10d922c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:16.879463 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.879328 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12282b71-4b77-497a-abac-9042f10d922c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84\" (UID: \"12282b71-4b77-497a-abac-9042f10d922c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:16.879463 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.879360 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12282b71-4b77-497a-abac-9042f10d922c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84\" (UID: \"12282b71-4b77-497a-abac-9042f10d922c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:16.879463 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.879410 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12282b71-4b77-497a-abac-9042f10d922c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84\" (UID: \"12282b71-4b77-497a-abac-9042f10d922c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:16.879647 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.879529 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12282b71-4b77-497a-abac-9042f10d922c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84\" (UID: \"12282b71-4b77-497a-abac-9042f10d922c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:16.879647 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.879554 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12282b71-4b77-497a-abac-9042f10d922c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84\" (UID: \"12282b71-4b77-497a-abac-9042f10d922c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:16.980159 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.980120 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12282b71-4b77-497a-abac-9042f10d922c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84\" (UID: \"12282b71-4b77-497a-abac-9042f10d922c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:16.980159 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.980161 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12282b71-4b77-497a-abac-9042f10d922c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84\" (UID: \"12282b71-4b77-497a-abac-9042f10d922c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:16.980426 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.980215 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12282b71-4b77-497a-abac-9042f10d922c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84\" (UID: \"12282b71-4b77-497a-abac-9042f10d922c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:16.980426 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.980232 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12282b71-4b77-497a-abac-9042f10d922c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84\" (UID: \"12282b71-4b77-497a-abac-9042f10d922c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:16.980426 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.980283 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6ppt\" (UniqueName: \"kubernetes.io/projected/12282b71-4b77-497a-abac-9042f10d922c-kube-api-access-c6ppt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84\" (UID: \"12282b71-4b77-497a-abac-9042f10d922c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:16.980426 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.980312 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12282b71-4b77-497a-abac-9042f10d922c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84\" (UID: \"12282b71-4b77-497a-abac-9042f10d922c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:16.980715 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.980692 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12282b71-4b77-497a-abac-9042f10d922c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84\" (UID: \"12282b71-4b77-497a-abac-9042f10d922c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:16.980772 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.980744 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12282b71-4b77-497a-abac-9042f10d922c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84\" (UID: \"12282b71-4b77-497a-abac-9042f10d922c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:16.980772 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.980756 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12282b71-4b77-497a-abac-9042f10d922c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84\" (UID: \"12282b71-4b77-497a-abac-9042f10d922c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:16.982560 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.982539 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12282b71-4b77-497a-abac-9042f10d922c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84\" (UID: \"12282b71-4b77-497a-abac-9042f10d922c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:16.982940 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.982920 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12282b71-4b77-497a-abac-9042f10d922c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84\" (UID: \"12282b71-4b77-497a-abac-9042f10d922c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:16.989572 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:16.989535 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6ppt\" (UniqueName: \"kubernetes.io/projected/12282b71-4b77-497a-abac-9042f10d922c-kube-api-access-c6ppt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84\" (UID: \"12282b71-4b77-497a-abac-9042f10d922c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:17.096291 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:17.096256 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:17.236753 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:17.236723 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84"] Jul 09 22:47:17.238397 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:47:17.238372 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12282b71_4b77_497a_abac_9042f10d922c.slice/crio-a3d12e62c4636101c71cc4eb0e93cc1fb1a9ab752611955b2250648f875aafa2 WatchSource:0}: Error finding container a3d12e62c4636101c71cc4eb0e93cc1fb1a9ab752611955b2250648f875aafa2: Status 404 returned error can't find the container with id a3d12e62c4636101c71cc4eb0e93cc1fb1a9ab752611955b2250648f875aafa2 Jul 09 22:47:17.653937 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:17.653898 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" event={"ID":"12282b71-4b77-497a-abac-9042f10d922c","Type":"ContainerStarted","Data":"21b0db920dead82e4d340c7e1c17ad39c5af625f8513fa2b11fbac1cd31b0518"} Jul 09 22:47:17.654126 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:17.653959 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" event={"ID":"12282b71-4b77-497a-abac-9042f10d922c","Type":"ContainerStarted","Data":"a3d12e62c4636101c71cc4eb0e93cc1fb1a9ab752611955b2250648f875aafa2"} Jul 09 22:47:17.675449 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:17.675346 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" podStartSLOduration=1.499239206 podStartE2EDuration="1.675326247s" totalImagesPullingTime="176.087041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:47:17.674626314 +0000 UTC m=+862.215705128" watchObservedRunningTime="2026-07-09 22:47:17.675326247 +0000 UTC m=+862.216405041" Jul 09 22:47:24.490533 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:24.490464 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:47:26.023221 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:26.023178 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:47:27.096986 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:27.096945 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:27.097402 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:27.097037 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:27.110023 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:27.109995 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:27.714996 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:27.714969 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84" Jul 09 22:47:33.183079 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:47:33.183038 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:48:11.922232 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:48:11.922194 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:48:16.169466 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:48:16.169428 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:48:16.265964 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:48:16.265927 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:48:17.375105 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:48:17.375072 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:48:17.506560 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:48:17.506523 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:48:21.928416 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:48:21.928383 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:48:22.024730 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:48:22.024684 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:48:46.828898 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:48:46.828859 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:48:46.932655 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:48:46.932616 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:49:28.212599 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:28.212508 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:49:29.882989 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:29.882946 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:49:29.995411 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:29.995373 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f9f7f5566-m4mbf"] Jul 09 22:49:29.998630 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:29.998614 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9f7f5566-m4mbf" Jul 09 22:49:30.013377 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:30.013342 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 22:49:30.013377 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:30.013362 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wdzbn\"" Jul 09 22:49:30.013614 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:30.013379 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 22:49:30.015885 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:30.015866 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f9f7f5566-m4mbf"] Jul 09 22:49:30.084332 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:30.084293 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f21b9db1-27c9-4415-83f6-fd1c8be98acd-webhook-cert\") pod \"maas-controller-7f9f7f5566-m4mbf\" (UID: \"f21b9db1-27c9-4415-83f6-fd1c8be98acd\") " pod="opendatahub/maas-controller-7f9f7f5566-m4mbf" Jul 09 22:49:30.084554 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:30.084350 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q2dj\" (UniqueName: \"kubernetes.io/projected/f21b9db1-27c9-4415-83f6-fd1c8be98acd-kube-api-access-6q2dj\") pod \"maas-controller-7f9f7f5566-m4mbf\" (UID: \"f21b9db1-27c9-4415-83f6-fd1c8be98acd\") " pod="opendatahub/maas-controller-7f9f7f5566-m4mbf" Jul 09 22:49:30.184975 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:30.184885 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f21b9db1-27c9-4415-83f6-fd1c8be98acd-webhook-cert\") pod \"maas-controller-7f9f7f5566-m4mbf\" (UID: \"f21b9db1-27c9-4415-83f6-fd1c8be98acd\") " pod="opendatahub/maas-controller-7f9f7f5566-m4mbf" Jul 09 22:49:30.184975 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:30.184933 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q2dj\" (UniqueName: \"kubernetes.io/projected/f21b9db1-27c9-4415-83f6-fd1c8be98acd-kube-api-access-6q2dj\") pod \"maas-controller-7f9f7f5566-m4mbf\" (UID: \"f21b9db1-27c9-4415-83f6-fd1c8be98acd\") " pod="opendatahub/maas-controller-7f9f7f5566-m4mbf" Jul 09 22:49:30.187377 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:30.187341 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f21b9db1-27c9-4415-83f6-fd1c8be98acd-webhook-cert\") pod \"maas-controller-7f9f7f5566-m4mbf\" (UID: \"f21b9db1-27c9-4415-83f6-fd1c8be98acd\") " pod="opendatahub/maas-controller-7f9f7f5566-m4mbf" Jul 09 22:49:30.194864 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:30.194837 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q2dj\" (UniqueName: \"kubernetes.io/projected/f21b9db1-27c9-4415-83f6-fd1c8be98acd-kube-api-access-6q2dj\") pod \"maas-controller-7f9f7f5566-m4mbf\" (UID: \"f21b9db1-27c9-4415-83f6-fd1c8be98acd\") " pod="opendatahub/maas-controller-7f9f7f5566-m4mbf" Jul 09 22:49:30.309716 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:30.309674 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9f7f5566-m4mbf" Jul 09 22:49:30.451939 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:30.451914 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f9f7f5566-m4mbf"] Jul 09 22:49:30.453188 ip-10-0-135-80 kubenswrapper[2568]: W0709 22:49:30.453155 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21b9db1_27c9_4415_83f6_fd1c8be98acd.slice/crio-ef85a4f17e83a8a1a15c5b033a988c149413fc49460fa500a1e5e052697aa941 WatchSource:0}: Error finding container ef85a4f17e83a8a1a15c5b033a988c149413fc49460fa500a1e5e052697aa941: Status 404 returned error can't find the container with id ef85a4f17e83a8a1a15c5b033a988c149413fc49460fa500a1e5e052697aa941 Jul 09 22:49:31.340792 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:31.340757 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9f7f5566-m4mbf" event={"ID":"f21b9db1-27c9-4415-83f6-fd1c8be98acd","Type":"ContainerStarted","Data":"a639572d6d6477603dc8090aa06b94f1349fc1e8d6bcff01f47625433a58e2cd"} Jul 09 22:49:31.341198 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:31.340816 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9f7f5566-m4mbf" event={"ID":"f21b9db1-27c9-4415-83f6-fd1c8be98acd","Type":"ContainerStarted","Data":"ef85a4f17e83a8a1a15c5b033a988c149413fc49460fa500a1e5e052697aa941"} Jul 09 22:49:31.341198 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:31.340848 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f9f7f5566-m4mbf" Jul 09 22:49:31.364925 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:31.364878 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f9f7f5566-m4mbf" podStartSLOduration=1.922064088 podStartE2EDuration="2.364863252s" totalImagesPullingTime="442.799164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:49:31.364721259 +0000 UTC m=+995.905800050" watchObservedRunningTime="2026-07-09 22:49:31.364863252 +0000 UTC m=+995.905942046" Jul 09 22:49:42.352456 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:49:42.352423 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f9f7f5566-m4mbf" Jul 09 22:50:09.186384 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:50:09.186322 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:50:09.276175 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:50:09.276120 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:50:09.410961 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:50:09.410923 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:50:14.573163 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:50:14.573123 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:50:15.461028 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:50:15.460977 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:50:15.534671 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:50:15.534635 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:50:15.650062 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:50:15.650021 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:50:15.845535 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:50:15.845473 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:50:22.163108 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:50:22.163068 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:50:22.261543 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:50:22.261477 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:54:20.726948 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:54:20.726915 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 22:54:55.928972 ip-10-0-135-80 kubenswrapper[2568]: I0709 22:54:55.928935 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:00:00.136497 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:00:00.136407 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727300-l8k9r"] Jul 09 23:00:00.139700 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:00:00.139684 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727300-l8k9r" Jul 09 23:00:00.142093 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:00:00.142069 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hn6dj\"" Jul 09 23:00:00.149465 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:00:00.149441 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727300-l8k9r" Jul 09 23:00:00.152293 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:00:00.152262 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727300-l8k9r"] Jul 09 23:00:00.289254 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:00:00.289224 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727300-l8k9r"] Jul 09 23:00:00.291105 ip-10-0-135-80 kubenswrapper[2568]: W0709 23:00:00.291081 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d516fa_220b_4d55_b69d_ec7aca75659c.slice/crio-d8ac269f298af66d34b000e6bf53ad3e2453bd6316b8ade9432b9fb085ed3479 WatchSource:0}: Error finding container d8ac269f298af66d34b000e6bf53ad3e2453bd6316b8ade9432b9fb085ed3479: Status 404 returned error can't find the container with id d8ac269f298af66d34b000e6bf53ad3e2453bd6316b8ade9432b9fb085ed3479 Jul 09 23:00:00.292827 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:00:00.292808 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 23:00:00.601657 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:00:00.601620 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727300-l8k9r" event={"ID":"38d516fa-220b-4d55-b69d-ec7aca75659c","Type":"ContainerStarted","Data":"d8ac269f298af66d34b000e6bf53ad3e2453bd6316b8ade9432b9fb085ed3479"} Jul 09 23:00:11.658829 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:00:11.658790 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727300-l8k9r" event={"ID":"38d516fa-220b-4d55-b69d-ec7aca75659c","Type":"ContainerStarted","Data":"9953b9cc24c2161ddedce3a44a6fe80a97f43938dd47283589c9dd8d023a1efe"} Jul 09 23:00:11.673307 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:00:11.673246 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727300-l8k9r" podStartSLOduration=1.308105622 podStartE2EDuration="11.67323043s" totalImagesPullingTime="10.365124808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:00:11.672115347 +0000 UTC m=+1636.213194142" watchObservedRunningTime="2026-07-09 23:00:11.67323043 +0000 UTC m=+1636.214309265" Jul 09 23:00:29.690186 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:00:29.690150 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:00.010215 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:00.010174 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727300-l8k9r"] Jul 09 23:01:00.010837 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:00.010428 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727300-l8k9r" podUID="38d516fa-220b-4d55-b69d-ec7aca75659c" containerName="cleanup" containerID="cri-o://9953b9cc24c2161ddedce3a44a6fe80a97f43938dd47283589c9dd8d023a1efe" gracePeriod=30 Jul 09 23:01:09.207931 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:09.207868 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:14.617459 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:14.617423 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:22.894407 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:22.894316 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:28.015749 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:28.015712 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:30.209208 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:30.209178 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727300-l8k9r" Jul 09 23:01:31.041642 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:31.041596 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:31.073788 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:31.073756 2568 generic.go:356] "Generic (PLEG): container finished" podID="38d516fa-220b-4d55-b69d-ec7aca75659c" containerID="9953b9cc24c2161ddedce3a44a6fe80a97f43938dd47283589c9dd8d023a1efe" exitCode=137 Jul 09 23:01:31.074015 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:31.073824 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727300-l8k9r" Jul 09 23:01:31.074015 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:31.073833 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727300-l8k9r" event={"ID":"38d516fa-220b-4d55-b69d-ec7aca75659c","Type":"ContainerDied","Data":"9953b9cc24c2161ddedce3a44a6fe80a97f43938dd47283589c9dd8d023a1efe"} Jul 09 23:01:31.074015 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:31.073888 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727300-l8k9r" event={"ID":"38d516fa-220b-4d55-b69d-ec7aca75659c","Type":"ContainerDied","Data":"d8ac269f298af66d34b000e6bf53ad3e2453bd6316b8ade9432b9fb085ed3479"} Jul 09 23:01:31.074015 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:31.073911 2568 scope.go:117] "RemoveContainer" containerID="9953b9cc24c2161ddedce3a44a6fe80a97f43938dd47283589c9dd8d023a1efe" Jul 09 23:01:31.083719 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:31.083701 2568 scope.go:117] "RemoveContainer" containerID="9953b9cc24c2161ddedce3a44a6fe80a97f43938dd47283589c9dd8d023a1efe" Jul 09 23:01:31.083992 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:01:31.083975 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9953b9cc24c2161ddedce3a44a6fe80a97f43938dd47283589c9dd8d023a1efe\": container with ID starting with 9953b9cc24c2161ddedce3a44a6fe80a97f43938dd47283589c9dd8d023a1efe not found: ID does not exist" containerID="9953b9cc24c2161ddedce3a44a6fe80a97f43938dd47283589c9dd8d023a1efe" Jul 09 23:01:31.084050 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:31.084001 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9953b9cc24c2161ddedce3a44a6fe80a97f43938dd47283589c9dd8d023a1efe"} err="failed to get container status \"9953b9cc24c2161ddedce3a44a6fe80a97f43938dd47283589c9dd8d023a1efe\": rpc error: code = NotFound desc = could not find container \"9953b9cc24c2161ddedce3a44a6fe80a97f43938dd47283589c9dd8d023a1efe\": container with ID starting with 9953b9cc24c2161ddedce3a44a6fe80a97f43938dd47283589c9dd8d023a1efe not found: ID does not exist" Jul 09 23:01:31.111128 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:31.111090 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727300-l8k9r"] Jul 09 23:01:31.117018 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:31.116978 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727300-l8k9r"] Jul 09 23:01:31.139647 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:31.139611 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:31.571568 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:31.571538 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:32.049964 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:32.049877 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38d516fa-220b-4d55-b69d-ec7aca75659c" path="/var/lib/kubelet/pods/38d516fa-220b-4d55-b69d-ec7aca75659c/volumes" Jul 09 23:01:37.407598 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:37.407562 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:37.473572 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:37.473535 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:37.622730 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:37.622694 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:37.690439 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:37.690301 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:38.737975 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:38.737939 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:38.837619 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:38.837582 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:38.935157 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:38.935124 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt"] Jul 09 23:01:38.935597 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:38.935578 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38d516fa-220b-4d55-b69d-ec7aca75659c" containerName="cleanup" Jul 09 23:01:38.935597 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:38.935596 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="38d516fa-220b-4d55-b69d-ec7aca75659c" containerName="cleanup" Jul 09 23:01:38.935752 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:38.935690 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="38d516fa-220b-4d55-b69d-ec7aca75659c" containerName="cleanup" Jul 09 23:01:38.940057 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:38.940033 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" Jul 09 23:01:38.942620 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:38.942598 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a1d7899b-serving-cert\"" Jul 09 23:01:38.952618 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:38.952586 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt"] Jul 09 23:01:38.967449 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:38.967413 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:39.028706 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:39.028611 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e755e85-daac-4b26-87c3-b72ca77dfbe5-maas-api-tls\") pod \"maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt\" (UID: \"0e755e85-daac-4b26-87c3-b72ca77dfbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" Jul 09 23:01:39.028895 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:39.028779 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e755e85-daac-4b26-87c3-b72ca77dfbe5-openshift-service-ca\") pod \"maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt\" (UID: \"0e755e85-daac-4b26-87c3-b72ca77dfbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" Jul 09 23:01:39.028895 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:39.028824 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsgt4\" (UniqueName: \"kubernetes.io/projected/0e755e85-daac-4b26-87c3-b72ca77dfbe5-kube-api-access-rsgt4\") pod \"maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt\" (UID: \"0e755e85-daac-4b26-87c3-b72ca77dfbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" Jul 09 23:01:39.129686 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:39.129651 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e755e85-daac-4b26-87c3-b72ca77dfbe5-openshift-service-ca\") pod \"maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt\" (UID: \"0e755e85-daac-4b26-87c3-b72ca77dfbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" Jul 09 23:01:39.129686 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:39.129684 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsgt4\" (UniqueName: \"kubernetes.io/projected/0e755e85-daac-4b26-87c3-b72ca77dfbe5-kube-api-access-rsgt4\") pod \"maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt\" (UID: \"0e755e85-daac-4b26-87c3-b72ca77dfbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" Jul 09 23:01:39.129958 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:39.129742 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e755e85-daac-4b26-87c3-b72ca77dfbe5-maas-api-tls\") pod \"maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt\" (UID: \"0e755e85-daac-4b26-87c3-b72ca77dfbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" Jul 09 23:01:39.129958 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:01:39.129862 2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-serving-cert: secret "maas-api-e2e-ait-a1d7899b-serving-cert" not found Jul 09 23:01:39.129958 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:01:39.129945 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e755e85-daac-4b26-87c3-b72ca77dfbe5-maas-api-tls podName:0e755e85-daac-4b26-87c3-b72ca77dfbe5 nodeName:}" failed. No retries permitted until 2026-07-09 23:01:39.629923152 +0000 UTC m=+1724.171001924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e755e85-daac-4b26-87c3-b72ca77dfbe5-maas-api-tls") pod "maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" (UID: "0e755e85-daac-4b26-87c3-b72ca77dfbe5") : secret "maas-api-e2e-ait-a1d7899b-serving-cert" not found Jul 09 23:01:39.130418 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:39.130396 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e755e85-daac-4b26-87c3-b72ca77dfbe5-openshift-service-ca\") pod \"maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt\" (UID: \"0e755e85-daac-4b26-87c3-b72ca77dfbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" Jul 09 23:01:39.139294 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:39.139265 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsgt4\" (UniqueName: \"kubernetes.io/projected/0e755e85-daac-4b26-87c3-b72ca77dfbe5-kube-api-access-rsgt4\") pod \"maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt\" (UID: \"0e755e85-daac-4b26-87c3-b72ca77dfbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" Jul 09 23:01:39.363882 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:39.363844 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:39.492996 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:39.492958 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:39.593848 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:39.593811 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:39.634302 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:39.634191 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e755e85-daac-4b26-87c3-b72ca77dfbe5-maas-api-tls\") pod \"maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt\" (UID: \"0e755e85-daac-4b26-87c3-b72ca77dfbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" Jul 09 23:01:39.637185 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:39.637139 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e755e85-daac-4b26-87c3-b72ca77dfbe5-maas-api-tls\") pod \"maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt\" (UID: \"0e755e85-daac-4b26-87c3-b72ca77dfbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" Jul 09 23:01:39.851886 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:39.851847 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" Jul 09 23:01:39.993734 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:39.993706 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt"] Jul 09 23:01:39.995532 ip-10-0-135-80 kubenswrapper[2568]: W0709 23:01:39.995481 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e755e85_daac_4b26_87c3_b72ca77dfbe5.slice/crio-7124bcc1632bce649700e2b835a175a21aed5a31c1073719b863ee8cdbb27903 WatchSource:0}: Error finding container 7124bcc1632bce649700e2b835a175a21aed5a31c1073719b863ee8cdbb27903: Status 404 returned error can't find the container with id 7124bcc1632bce649700e2b835a175a21aed5a31c1073719b863ee8cdbb27903 Jul 09 23:01:40.124301 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:40.124265 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" event={"ID":"0e755e85-daac-4b26-87c3-b72ca77dfbe5","Type":"ContainerStarted","Data":"7124bcc1632bce649700e2b835a175a21aed5a31c1073719b863ee8cdbb27903"} Jul 09 23:01:41.131984 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:41.131944 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" event={"ID":"0e755e85-daac-4b26-87c3-b72ca77dfbe5","Type":"ContainerStarted","Data":"9eb6270ce9405de8d78ce98819c94682770619e5cae9607e9fd6f0a11d74b817"} Jul 09 23:01:41.132402 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:41.132012 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" Jul 09 23:01:41.153417 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:41.153345 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" podStartSLOduration=2.597702421 podStartE2EDuration="3.153324057s" totalImagesPullingTime="555.621636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:01:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:01:41.150137448 +0000 UTC m=+1725.691216268" watchObservedRunningTime="2026-07-09 23:01:41.153324057 +0000 UTC m=+1725.694402853" Jul 09 23:01:41.178728 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:41.178675 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:43.868382 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:43.868345 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:44.197619 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:44.197523 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt"] Jul 09 23:01:44.197809 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:44.197743 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" podUID="0e755e85-daac-4b26-87c3-b72ca77dfbe5" containerName="maas-api" containerID="cri-o://9eb6270ce9405de8d78ce98819c94682770619e5cae9607e9fd6f0a11d74b817" gracePeriod=30 Jul 09 23:01:44.197809 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:44.197755 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:44.298606 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:44.298568 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:44.446315 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:44.446281 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" Jul 09 23:01:44.480887 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:44.480794 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e755e85-daac-4b26-87c3-b72ca77dfbe5-maas-api-tls\") pod \"0e755e85-daac-4b26-87c3-b72ca77dfbe5\" (UID: \"0e755e85-daac-4b26-87c3-b72ca77dfbe5\") " Jul 09 23:01:44.480887 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:44.480836 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsgt4\" (UniqueName: \"kubernetes.io/projected/0e755e85-daac-4b26-87c3-b72ca77dfbe5-kube-api-access-rsgt4\") pod \"0e755e85-daac-4b26-87c3-b72ca77dfbe5\" (UID: \"0e755e85-daac-4b26-87c3-b72ca77dfbe5\") " Jul 09 23:01:44.480887 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:44.480866 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e755e85-daac-4b26-87c3-b72ca77dfbe5-openshift-service-ca\") pod \"0e755e85-daac-4b26-87c3-b72ca77dfbe5\" (UID: \"0e755e85-daac-4b26-87c3-b72ca77dfbe5\") " Jul 09 23:01:44.481265 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:44.481243 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e755e85-daac-4b26-87c3-b72ca77dfbe5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0e755e85-daac-4b26-87c3-b72ca77dfbe5" (UID: "0e755e85-daac-4b26-87c3-b72ca77dfbe5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:01:44.483148 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:44.483117 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e755e85-daac-4b26-87c3-b72ca77dfbe5-kube-api-access-rsgt4" (OuterVolumeSpecName: "kube-api-access-rsgt4") pod "0e755e85-daac-4b26-87c3-b72ca77dfbe5" (UID: "0e755e85-daac-4b26-87c3-b72ca77dfbe5"). InnerVolumeSpecName "kube-api-access-rsgt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:01:44.483148 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:44.483126 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e755e85-daac-4b26-87c3-b72ca77dfbe5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0e755e85-daac-4b26-87c3-b72ca77dfbe5" (UID: "0e755e85-daac-4b26-87c3-b72ca77dfbe5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:01:44.582128 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:44.582071 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e755e85-daac-4b26-87c3-b72ca77dfbe5-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:01:44.582128 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:44.582119 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsgt4\" (UniqueName: \"kubernetes.io/projected/0e755e85-daac-4b26-87c3-b72ca77dfbe5-kube-api-access-rsgt4\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:01:44.582128 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:44.582135 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e755e85-daac-4b26-87c3-b72ca77dfbe5-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:01:45.155279 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:45.155238 2568 generic.go:356] "Generic (PLEG): container finished" podID="0e755e85-daac-4b26-87c3-b72ca77dfbe5" containerID="9eb6270ce9405de8d78ce98819c94682770619e5cae9607e9fd6f0a11d74b817" exitCode=0 Jul 09 23:01:45.155678 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:45.155300 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" Jul 09 23:01:45.155678 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:45.155314 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" event={"ID":"0e755e85-daac-4b26-87c3-b72ca77dfbe5","Type":"ContainerDied","Data":"9eb6270ce9405de8d78ce98819c94682770619e5cae9607e9fd6f0a11d74b817"} Jul 09 23:01:45.155678 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:45.155373 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt" event={"ID":"0e755e85-daac-4b26-87c3-b72ca77dfbe5","Type":"ContainerDied","Data":"7124bcc1632bce649700e2b835a175a21aed5a31c1073719b863ee8cdbb27903"} Jul 09 23:01:45.155678 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:45.155398 2568 scope.go:117] "RemoveContainer" containerID="9eb6270ce9405de8d78ce98819c94682770619e5cae9607e9fd6f0a11d74b817" Jul 09 23:01:45.164541 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:45.164513 2568 scope.go:117] "RemoveContainer" containerID="9eb6270ce9405de8d78ce98819c94682770619e5cae9607e9fd6f0a11d74b817" Jul 09 23:01:45.164904 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:01:45.164873 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eb6270ce9405de8d78ce98819c94682770619e5cae9607e9fd6f0a11d74b817\": container with ID starting with 9eb6270ce9405de8d78ce98819c94682770619e5cae9607e9fd6f0a11d74b817 not found: ID does not exist" containerID="9eb6270ce9405de8d78ce98819c94682770619e5cae9607e9fd6f0a11d74b817" Jul 09 23:01:45.164977 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:45.164905 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eb6270ce9405de8d78ce98819c94682770619e5cae9607e9fd6f0a11d74b817"} err="failed to get container status \"9eb6270ce9405de8d78ce98819c94682770619e5cae9607e9fd6f0a11d74b817\": rpc error: code = NotFound desc = could not find container \"9eb6270ce9405de8d78ce98819c94682770619e5cae9607e9fd6f0a11d74b817\": container with ID starting with 9eb6270ce9405de8d78ce98819c94682770619e5cae9607e9fd6f0a11d74b817 not found: ID does not exist" Jul 09 23:01:45.176592 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:45.176559 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt"] Jul 09 23:01:45.180863 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:45.180823 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d7899b-55db8cc8bc-tfmvt"] Jul 09 23:01:46.048680 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:46.048645 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e755e85-daac-4b26-87c3-b72ca77dfbe5" path="/var/lib/kubelet/pods/0e755e85-daac-4b26-87c3-b72ca77dfbe5/volumes" Jul 09 23:01:47.952383 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:47.952351 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:49.896151 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:49.896112 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:49.997349 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:49.997311 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:50.198797 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:50.198707 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:50.298893 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:50.298850 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:51.211651 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.211614 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:51.277902 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.277867 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj"] Jul 09 23:01:51.278453 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.278434 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e755e85-daac-4b26-87c3-b72ca77dfbe5" containerName="maas-api" Jul 09 23:01:51.278530 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.278458 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e755e85-daac-4b26-87c3-b72ca77dfbe5" containerName="maas-api" Jul 09 23:01:51.278580 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.278557 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e755e85-daac-4b26-87c3-b72ca77dfbe5" containerName="maas-api" Jul 09 23:01:51.283370 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.283338 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" Jul 09 23:01:51.286015 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.285985 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-201d22c6-serving-cert\"" Jul 09 23:01:51.292028 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.291773 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj"] Jul 09 23:01:51.292816 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.292784 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:51.340536 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.340465 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0149208c-44cc-45b8-964e-b9df30cdc010-openshift-service-ca\") pod \"maas-api-e2e-ait-201d22c6-5544955556-2hxbj\" (UID: \"0149208c-44cc-45b8-964e-b9df30cdc010\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" Jul 09 23:01:51.340757 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.340556 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0149208c-44cc-45b8-964e-b9df30cdc010-maas-api-tls\") pod \"maas-api-e2e-ait-201d22c6-5544955556-2hxbj\" (UID: \"0149208c-44cc-45b8-964e-b9df30cdc010\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" Jul 09 23:01:51.340757 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.340616 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2v9j\" (UniqueName: \"kubernetes.io/projected/0149208c-44cc-45b8-964e-b9df30cdc010-kube-api-access-x2v9j\") pod \"maas-api-e2e-ait-201d22c6-5544955556-2hxbj\" (UID: \"0149208c-44cc-45b8-964e-b9df30cdc010\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" Jul 09 23:01:51.359386 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.359355 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:51.441583 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.441534 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0149208c-44cc-45b8-964e-b9df30cdc010-openshift-service-ca\") pod \"maas-api-e2e-ait-201d22c6-5544955556-2hxbj\" (UID: \"0149208c-44cc-45b8-964e-b9df30cdc010\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" Jul 09 23:01:51.441583 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.441588 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0149208c-44cc-45b8-964e-b9df30cdc010-maas-api-tls\") pod \"maas-api-e2e-ait-201d22c6-5544955556-2hxbj\" (UID: \"0149208c-44cc-45b8-964e-b9df30cdc010\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" Jul 09 23:01:51.441859 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.441644 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2v9j\" (UniqueName: \"kubernetes.io/projected/0149208c-44cc-45b8-964e-b9df30cdc010-kube-api-access-x2v9j\") pod \"maas-api-e2e-ait-201d22c6-5544955556-2hxbj\" (UID: \"0149208c-44cc-45b8-964e-b9df30cdc010\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" Jul 09 23:01:51.442242 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.442212 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0149208c-44cc-45b8-964e-b9df30cdc010-openshift-service-ca\") pod \"maas-api-e2e-ait-201d22c6-5544955556-2hxbj\" (UID: \"0149208c-44cc-45b8-964e-b9df30cdc010\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" Jul 09 23:01:51.444134 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.444108 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0149208c-44cc-45b8-964e-b9df30cdc010-maas-api-tls\") pod \"maas-api-e2e-ait-201d22c6-5544955556-2hxbj\" (UID: \"0149208c-44cc-45b8-964e-b9df30cdc010\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" Jul 09 23:01:51.450775 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.450745 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2v9j\" (UniqueName: \"kubernetes.io/projected/0149208c-44cc-45b8-964e-b9df30cdc010-kube-api-access-x2v9j\") pod \"maas-api-e2e-ait-201d22c6-5544955556-2hxbj\" (UID: \"0149208c-44cc-45b8-964e-b9df30cdc010\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" Jul 09 23:01:51.472407 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.472326 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:51.596885 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.596841 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" Jul 09 23:01:51.742877 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.742849 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj"] Jul 09 23:01:51.744786 ip-10-0-135-80 kubenswrapper[2568]: W0709 23:01:51.744754 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0149208c_44cc_45b8_964e_b9df30cdc010.slice/crio-95fa7f723eafe1e3e7c240b709a9d691c973810ed4289c41d3795dd4cefd49da WatchSource:0}: Error finding container 95fa7f723eafe1e3e7c240b709a9d691c973810ed4289c41d3795dd4cefd49da: Status 404 returned error can't find the container with id 95fa7f723eafe1e3e7c240b709a9d691c973810ed4289c41d3795dd4cefd49da Jul 09 23:01:51.870245 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.870201 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:51.971089 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:51.971034 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:52.196531 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:52.196454 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" event={"ID":"0149208c-44cc-45b8-964e-b9df30cdc010","Type":"ContainerStarted","Data":"787a560dd001fa78535ad138a875b8af43c62b9903eef4c92992b021fe873b87"} Jul 09 23:01:52.196738 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:52.196540 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" Jul 09 23:01:52.196738 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:52.196564 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" event={"ID":"0149208c-44cc-45b8-964e-b9df30cdc010","Type":"ContainerStarted","Data":"95fa7f723eafe1e3e7c240b709a9d691c973810ed4289c41d3795dd4cefd49da"} Jul 09 23:01:52.216730 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:52.216668 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" podStartSLOduration=0.847690601 podStartE2EDuration="1.21664568s" totalImagesPullingTime="368.955079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:01:52.214372027 +0000 UTC m=+1736.755450844" watchObservedRunningTime="2026-07-09 23:01:52.21664568 +0000 UTC m=+1736.757724476" Jul 09 23:01:54.309610 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:54.309579 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:56.383510 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:56.383387 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:56.862000 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:56.861961 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj"] Jul 09 23:01:56.862218 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:56.862169 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" podUID="0149208c-44cc-45b8-964e-b9df30cdc010" containerName="maas-api" containerID="cri-o://787a560dd001fa78535ad138a875b8af43c62b9903eef4c92992b021fe873b87" gracePeriod=30 Jul 09 23:01:56.880268 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:56.880209 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:01:57.115155 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:57.115011 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" Jul 09 23:01:57.200323 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:57.200282 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2v9j\" (UniqueName: \"kubernetes.io/projected/0149208c-44cc-45b8-964e-b9df30cdc010-kube-api-access-x2v9j\") pod \"0149208c-44cc-45b8-964e-b9df30cdc010\" (UID: \"0149208c-44cc-45b8-964e-b9df30cdc010\") " Jul 09 23:01:57.200323 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:57.200328 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0149208c-44cc-45b8-964e-b9df30cdc010-openshift-service-ca\") pod \"0149208c-44cc-45b8-964e-b9df30cdc010\" (UID: \"0149208c-44cc-45b8-964e-b9df30cdc010\") " Jul 09 23:01:57.200640 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:57.200437 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0149208c-44cc-45b8-964e-b9df30cdc010-maas-api-tls\") pod \"0149208c-44cc-45b8-964e-b9df30cdc010\" (UID: \"0149208c-44cc-45b8-964e-b9df30cdc010\") " Jul 09 23:01:57.200788 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:57.200745 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0149208c-44cc-45b8-964e-b9df30cdc010-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0149208c-44cc-45b8-964e-b9df30cdc010" (UID: "0149208c-44cc-45b8-964e-b9df30cdc010"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:01:57.202673 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:57.202636 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0149208c-44cc-45b8-964e-b9df30cdc010-kube-api-access-x2v9j" (OuterVolumeSpecName: "kube-api-access-x2v9j") pod "0149208c-44cc-45b8-964e-b9df30cdc010" (UID: "0149208c-44cc-45b8-964e-b9df30cdc010"). InnerVolumeSpecName "kube-api-access-x2v9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:01:57.202828 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:57.202736 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0149208c-44cc-45b8-964e-b9df30cdc010-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0149208c-44cc-45b8-964e-b9df30cdc010" (UID: "0149208c-44cc-45b8-964e-b9df30cdc010"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:01:57.224867 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:57.224832 2568 generic.go:356] "Generic (PLEG): container finished" podID="0149208c-44cc-45b8-964e-b9df30cdc010" containerID="787a560dd001fa78535ad138a875b8af43c62b9903eef4c92992b021fe873b87" exitCode=0 Jul 09 23:01:57.225077 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:57.224883 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" Jul 09 23:01:57.225077 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:57.224908 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" event={"ID":"0149208c-44cc-45b8-964e-b9df30cdc010","Type":"ContainerDied","Data":"787a560dd001fa78535ad138a875b8af43c62b9903eef4c92992b021fe873b87"} Jul 09 23:01:57.225077 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:57.224959 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj" event={"ID":"0149208c-44cc-45b8-964e-b9df30cdc010","Type":"ContainerDied","Data":"95fa7f723eafe1e3e7c240b709a9d691c973810ed4289c41d3795dd4cefd49da"} Jul 09 23:01:57.225077 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:57.224984 2568 scope.go:117] "RemoveContainer" containerID="787a560dd001fa78535ad138a875b8af43c62b9903eef4c92992b021fe873b87" Jul 09 23:01:57.240078 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:57.240057 2568 scope.go:117] "RemoveContainer" containerID="787a560dd001fa78535ad138a875b8af43c62b9903eef4c92992b021fe873b87" Jul 09 23:01:57.240388 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:01:57.240367 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"787a560dd001fa78535ad138a875b8af43c62b9903eef4c92992b021fe873b87\": container with ID starting with 787a560dd001fa78535ad138a875b8af43c62b9903eef4c92992b021fe873b87 not found: ID does not exist" containerID="787a560dd001fa78535ad138a875b8af43c62b9903eef4c92992b021fe873b87" Jul 09 23:01:57.240446 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:57.240397 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787a560dd001fa78535ad138a875b8af43c62b9903eef4c92992b021fe873b87"} err="failed to get container status \"787a560dd001fa78535ad138a875b8af43c62b9903eef4c92992b021fe873b87\": rpc error: code = NotFound desc = could not find container \"787a560dd001fa78535ad138a875b8af43c62b9903eef4c92992b021fe873b87\": container with ID starting with 787a560dd001fa78535ad138a875b8af43c62b9903eef4c92992b021fe873b87 not found: ID does not exist" Jul 09 23:01:57.254056 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:57.254021 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj"] Jul 09 23:01:57.260232 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:57.260198 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-201d22c6-5544955556-2hxbj"] Jul 09 23:01:57.301438 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:57.301393 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0149208c-44cc-45b8-964e-b9df30cdc010-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:01:57.301438 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:57.301430 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2v9j\" (UniqueName: \"kubernetes.io/projected/0149208c-44cc-45b8-964e-b9df30cdc010-kube-api-access-x2v9j\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:01:57.301438 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:57.301443 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0149208c-44cc-45b8-964e-b9df30cdc010-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:01:58.048189 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:01:58.048154 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0149208c-44cc-45b8-964e-b9df30cdc010" path="/var/lib/kubelet/pods/0149208c-44cc-45b8-964e-b9df30cdc010/volumes" Jul 09 23:02:02.507467 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:02.507428 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:02.615798 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:02.615753 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:05.410224 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:05.410169 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:08.188543 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.188474 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:08.286356 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.286314 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:08.381798 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.381752 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9"] Jul 09 23:02:08.382149 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.382133 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0149208c-44cc-45b8-964e-b9df30cdc010" containerName="maas-api" Jul 09 23:02:08.382201 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.382151 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0149208c-44cc-45b8-964e-b9df30cdc010" containerName="maas-api" Jul 09 23:02:08.382243 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.382208 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0149208c-44cc-45b8-964e-b9df30cdc010" containerName="maas-api" Jul 09 23:02:08.385410 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.385390 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.388461 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.388428 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-202791fd-openshift-default-dockercfg-tnd75\"" Jul 09 23:02:08.399578 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.399541 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9"] Jul 09 23:02:08.413179 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.413136 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:08.479616 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.479524 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:08.500782 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.500736 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-workload-certs\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.500782 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.500789 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-credential-socket\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.501021 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.500811 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmnn4\" (UniqueName: \"kubernetes.io/projected/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-kube-api-access-xmnn4\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.501021 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.500887 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-podinfo\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.501021 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.500963 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istiod-ca-cert\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.501021 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.500989 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-token\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.501021 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.501017 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-envoy\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.501310 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.501045 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-data\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.501310 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.501146 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-workload-socket\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.602079 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.602044 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-workload-certs\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.602296 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.602090 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-credential-socket\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.602296 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.602113 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmnn4\" (UniqueName: \"kubernetes.io/projected/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-kube-api-access-xmnn4\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.602296 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.602145 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-podinfo\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.602296 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.602170 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istiod-ca-cert\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.602296 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.602191 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-token\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.602296 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.602214 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-envoy\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.602296 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.602254 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-data\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.602699 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.602311 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-workload-socket\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.602699 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.602579 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-credential-socket\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.602699 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.602596 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-workload-certs\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.602859 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.602737 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-workload-socket\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.602859 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.602821 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-data\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.603132 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.603107 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istiod-ca-cert\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.604682 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.604665 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-podinfo\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.604765 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.604691 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-envoy\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.610117 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.610087 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-token\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.610959 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.610935 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmnn4\" (UniqueName: \"kubernetes.io/projected/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-kube-api-access-xmnn4\") pod \"e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.697091 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.697051 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:08.838351 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.838252 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9"] Jul 09 23:02:08.840756 ip-10-0-135-80 kubenswrapper[2568]: W0709 23:02:08.840732 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5abdaf7_4d6c_4473_b4b5_73246b90d9aa.slice/crio-ae340fbfe15779cd5863f79dd97df4f7fb55fe693b3a22a418d44401b3e7da6f WatchSource:0}: Error finding container ae340fbfe15779cd5863f79dd97df4f7fb55fe693b3a22a418d44401b3e7da6f: Status 404 returned error can't find the container with id ae340fbfe15779cd5863f79dd97df4f7fb55fe693b3a22a418d44401b3e7da6f Jul 09 23:02:08.843238 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.843202 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:02:08.843330 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.843275 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:02:08.843330 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:08.843306 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:02:09.291274 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.291232 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" event={"ID":"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa","Type":"ContainerStarted","Data":"b4c950ca7504acea6c49a3ef716b8b0d8549f64a4e7294692cdcd2f4f47d82f7"} Jul 09 23:02:09.291712 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.291286 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" event={"ID":"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa","Type":"ContainerStarted","Data":"ae340fbfe15779cd5863f79dd97df4f7fb55fe693b3a22a418d44401b3e7da6f"} Jul 09 23:02:09.312300 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.312244 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" podStartSLOduration=1.312225033 podStartE2EDuration="1.312225033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:02:09.309799077 +0000 UTC m=+1753.850877895" watchObservedRunningTime="2026-07-09 23:02:09.312225033 +0000 UTC m=+1753.853303827" Jul 09 23:02:09.599069 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.599034 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:09.686639 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.686601 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55"] Jul 09 23:02:09.687244 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.687206 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:09.690600 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.690575 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" Jul 09 23:02:09.693261 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.693235 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-202791fd-serving-cert\"" Jul 09 23:02:09.698577 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.698505 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:09.701121 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.699477 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55"] Jul 09 23:02:09.701121 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.700432 2568 patch_prober.go:28] interesting pod/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 09 23:02:09.701121 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.700483 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" podUID="f5abdaf7-4d6c-4473-b4b5-73246b90d9aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 09 23:02:09.753164 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.753120 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:09.813971 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.813932 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a4b0e3e-f271-4a48-9e01-e7a562ced4c1-maas-api-tls\") pod \"maas-api-e2e-derive-202791fd-56bb7576f8-hrd55\" (UID: \"1a4b0e3e-f271-4a48-9e01-e7a562ced4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" Jul 09 23:02:09.814179 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.813989 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ddr4\" (UniqueName: \"kubernetes.io/projected/1a4b0e3e-f271-4a48-9e01-e7a562ced4c1-kube-api-access-6ddr4\") pod \"maas-api-e2e-derive-202791fd-56bb7576f8-hrd55\" (UID: \"1a4b0e3e-f271-4a48-9e01-e7a562ced4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" Jul 09 23:02:09.814179 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.814108 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4b0e3e-f271-4a48-9e01-e7a562ced4c1-openshift-service-ca\") pod \"maas-api-e2e-derive-202791fd-56bb7576f8-hrd55\" (UID: \"1a4b0e3e-f271-4a48-9e01-e7a562ced4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" Jul 09 23:02:09.889259 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.889169 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:09.915290 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.915248 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4b0e3e-f271-4a48-9e01-e7a562ced4c1-openshift-service-ca\") pod \"maas-api-e2e-derive-202791fd-56bb7576f8-hrd55\" (UID: \"1a4b0e3e-f271-4a48-9e01-e7a562ced4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" Jul 09 23:02:09.915477 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.915408 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a4b0e3e-f271-4a48-9e01-e7a562ced4c1-maas-api-tls\") pod \"maas-api-e2e-derive-202791fd-56bb7576f8-hrd55\" (UID: \"1a4b0e3e-f271-4a48-9e01-e7a562ced4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" Jul 09 23:02:09.915477 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.915435 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ddr4\" (UniqueName: \"kubernetes.io/projected/1a4b0e3e-f271-4a48-9e01-e7a562ced4c1-kube-api-access-6ddr4\") pod \"maas-api-e2e-derive-202791fd-56bb7576f8-hrd55\" (UID: \"1a4b0e3e-f271-4a48-9e01-e7a562ced4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" Jul 09 23:02:09.916141 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.916116 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4b0e3e-f271-4a48-9e01-e7a562ced4c1-openshift-service-ca\") pod \"maas-api-e2e-derive-202791fd-56bb7576f8-hrd55\" (UID: \"1a4b0e3e-f271-4a48-9e01-e7a562ced4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" Jul 09 23:02:09.917919 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.917894 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a4b0e3e-f271-4a48-9e01-e7a562ced4c1-maas-api-tls\") pod \"maas-api-e2e-derive-202791fd-56bb7576f8-hrd55\" (UID: \"1a4b0e3e-f271-4a48-9e01-e7a562ced4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" Jul 09 23:02:09.924666 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:09.924630 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ddr4\" (UniqueName: \"kubernetes.io/projected/1a4b0e3e-f271-4a48-9e01-e7a562ced4c1-kube-api-access-6ddr4\") pod \"maas-api-e2e-derive-202791fd-56bb7576f8-hrd55\" (UID: \"1a4b0e3e-f271-4a48-9e01-e7a562ced4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" Jul 09 23:02:10.005665 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:10.005624 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" Jul 09 23:02:10.175835 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:10.175626 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55"] Jul 09 23:02:10.176083 ip-10-0-135-80 kubenswrapper[2568]: W0709 23:02:10.176043 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4b0e3e_f271_4a48_9e01_e7a562ced4c1.slice/crio-89f62b345826273416c952aa9dba86443de43cdbc2bb5ec277022f9474498fb8 WatchSource:0}: Error finding container 89f62b345826273416c952aa9dba86443de43cdbc2bb5ec277022f9474498fb8: Status 404 returned error can't find the container with id 89f62b345826273416c952aa9dba86443de43cdbc2bb5ec277022f9474498fb8 Jul 09 23:02:10.235168 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:10.235133 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:10.298263 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:10.298231 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" event={"ID":"1a4b0e3e-f271-4a48-9e01-e7a562ced4c1","Type":"ContainerStarted","Data":"89f62b345826273416c952aa9dba86443de43cdbc2bb5ec277022f9474498fb8"} Jul 09 23:02:10.335237 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:10.335204 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:10.701691 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:10.701657 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:11.304916 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:11.304876 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" event={"ID":"1a4b0e3e-f271-4a48-9e01-e7a562ced4c1","Type":"ContainerStarted","Data":"36f4ece1ff2d1d81014f3494c47b37735a29b87db3104272506d84cb2e214399"} Jul 09 23:02:11.305415 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:11.305087 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" Jul 09 23:02:11.305415 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:11.305109 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:11.306039 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:11.306021 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:11.323509 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:11.323434 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" podStartSLOduration=1.924405983 podStartE2EDuration="2.323415847s" totalImagesPullingTime="399.009864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:02:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:02:11.3223285 +0000 UTC m=+1755.863407294" watchObservedRunningTime="2026-07-09 23:02:11.323415847 +0000 UTC m=+1755.864494648" Jul 09 23:02:11.855107 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:11.855075 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:14.158038 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:14.158003 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9"] Jul 09 23:02:14.319831 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:14.319788 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" podUID="f5abdaf7-4d6c-4473-b4b5-73246b90d9aa" containerName="istio-proxy" containerID="cri-o://b4c950ca7504acea6c49a3ef716b8b0d8549f64a4e7294692cdcd2f4f47d82f7" gracePeriod=30 Jul 09 23:02:14.350306 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:14.350267 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55"] Jul 09 23:02:14.350618 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:14.350475 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" podUID="1a4b0e3e-f271-4a48-9e01-e7a562ced4c1" containerName="maas-api" containerID="cri-o://36f4ece1ff2d1d81014f3494c47b37735a29b87db3104272506d84cb2e214399" gracePeriod=30 Jul 09 23:02:14.638774 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:14.638742 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" Jul 09 23:02:14.685836 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:14.685795 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a4b0e3e-f271-4a48-9e01-e7a562ced4c1-maas-api-tls\") pod \"1a4b0e3e-f271-4a48-9e01-e7a562ced4c1\" (UID: \"1a4b0e3e-f271-4a48-9e01-e7a562ced4c1\") " Jul 09 23:02:14.686023 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:14.685912 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ddr4\" (UniqueName: \"kubernetes.io/projected/1a4b0e3e-f271-4a48-9e01-e7a562ced4c1-kube-api-access-6ddr4\") pod \"1a4b0e3e-f271-4a48-9e01-e7a562ced4c1\" (UID: \"1a4b0e3e-f271-4a48-9e01-e7a562ced4c1\") " Jul 09 23:02:14.686023 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:14.685962 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4b0e3e-f271-4a48-9e01-e7a562ced4c1-openshift-service-ca\") pod \"1a4b0e3e-f271-4a48-9e01-e7a562ced4c1\" (UID: \"1a4b0e3e-f271-4a48-9e01-e7a562ced4c1\") " Jul 09 23:02:14.686647 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:14.686605 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4b0e3e-f271-4a48-9e01-e7a562ced4c1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1a4b0e3e-f271-4a48-9e01-e7a562ced4c1" (UID: "1a4b0e3e-f271-4a48-9e01-e7a562ced4c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:02:14.688629 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:14.688597 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4b0e3e-f271-4a48-9e01-e7a562ced4c1-kube-api-access-6ddr4" (OuterVolumeSpecName: "kube-api-access-6ddr4") pod "1a4b0e3e-f271-4a48-9e01-e7a562ced4c1" (UID: "1a4b0e3e-f271-4a48-9e01-e7a562ced4c1"). InnerVolumeSpecName "kube-api-access-6ddr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:02:14.688741 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:14.688593 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a4b0e3e-f271-4a48-9e01-e7a562ced4c1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1a4b0e3e-f271-4a48-9e01-e7a562ced4c1" (UID: "1a4b0e3e-f271-4a48-9e01-e7a562ced4c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:02:14.787182 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:14.787074 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ddr4\" (UniqueName: \"kubernetes.io/projected/1a4b0e3e-f271-4a48-9e01-e7a562ced4c1-kube-api-access-6ddr4\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:02:14.787182 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:14.787105 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4b0e3e-f271-4a48-9e01-e7a562ced4c1-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:02:14.787182 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:14.787115 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a4b0e3e-f271-4a48-9e01-e7a562ced4c1-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:02:15.326641 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:15.326600 2568 generic.go:356] "Generic (PLEG): container finished" podID="1a4b0e3e-f271-4a48-9e01-e7a562ced4c1" containerID="36f4ece1ff2d1d81014f3494c47b37735a29b87db3104272506d84cb2e214399" exitCode=0 Jul 09 23:02:15.327078 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:15.326668 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" Jul 09 23:02:15.327078 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:15.326693 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" event={"ID":"1a4b0e3e-f271-4a48-9e01-e7a562ced4c1","Type":"ContainerDied","Data":"36f4ece1ff2d1d81014f3494c47b37735a29b87db3104272506d84cb2e214399"} Jul 09 23:02:15.327078 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:15.326750 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55" event={"ID":"1a4b0e3e-f271-4a48-9e01-e7a562ced4c1","Type":"ContainerDied","Data":"89f62b345826273416c952aa9dba86443de43cdbc2bb5ec277022f9474498fb8"} Jul 09 23:02:15.327078 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:15.326773 2568 scope.go:117] "RemoveContainer" containerID="36f4ece1ff2d1d81014f3494c47b37735a29b87db3104272506d84cb2e214399" Jul 09 23:02:15.336230 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:15.336208 2568 scope.go:117] "RemoveContainer" containerID="36f4ece1ff2d1d81014f3494c47b37735a29b87db3104272506d84cb2e214399" Jul 09 23:02:15.336562 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:02:15.336535 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f4ece1ff2d1d81014f3494c47b37735a29b87db3104272506d84cb2e214399\": container with ID starting with 36f4ece1ff2d1d81014f3494c47b37735a29b87db3104272506d84cb2e214399 not found: ID does not exist" containerID="36f4ece1ff2d1d81014f3494c47b37735a29b87db3104272506d84cb2e214399" Jul 09 23:02:15.336672 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:15.336571 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f4ece1ff2d1d81014f3494c47b37735a29b87db3104272506d84cb2e214399"} err="failed to get container status \"36f4ece1ff2d1d81014f3494c47b37735a29b87db3104272506d84cb2e214399\": rpc error: code = NotFound desc = could not find container \"36f4ece1ff2d1d81014f3494c47b37735a29b87db3104272506d84cb2e214399\": container with ID starting with 36f4ece1ff2d1d81014f3494c47b37735a29b87db3104272506d84cb2e214399 not found: ID does not exist" Jul 09 23:02:15.348520 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:15.348469 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55"] Jul 09 23:02:15.354380 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:15.354346 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-202791fd-56bb7576f8-hrd55"] Jul 09 23:02:15.464422 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:15.464371 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:15.511642 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:15.511592 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:15.687432 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:15.687395 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:16.048878 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:16.048798 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a4b0e3e-f271-4a48-9e01-e7a562ced4c1" path="/var/lib/kubelet/pods/1a4b0e3e-f271-4a48-9e01-e7a562ced4c1/volumes" Jul 09 23:02:20.031836 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.031801 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:20.132343 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.132316 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:20.278191 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.278165 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:20.337111 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.337079 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-workload-certs\") pod \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " Jul 09 23:02:20.337385 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.337123 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-workload-socket\") pod \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " Jul 09 23:02:20.337385 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.337165 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istiod-ca-cert\") pod \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " Jul 09 23:02:20.337385 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.337181 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-token\") pod \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " Jul 09 23:02:20.337385 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.337217 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmnn4\" (UniqueName: \"kubernetes.io/projected/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-kube-api-access-xmnn4\") pod \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " Jul 09 23:02:20.337385 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.337254 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-envoy\") pod \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " Jul 09 23:02:20.337720 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.337403 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-data\") pod \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " Jul 09 23:02:20.337720 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.337414 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f5abdaf7-4d6c-4473-b4b5-73246b90d9aa" (UID: "f5abdaf7-4d6c-4473-b4b5-73246b90d9aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:02:20.337720 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.337521 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-podinfo\") pod \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " Jul 09 23:02:20.337720 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.337554 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-credential-socket\") pod \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\" (UID: \"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa\") " Jul 09 23:02:20.337720 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.337585 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f5abdaf7-4d6c-4473-b4b5-73246b90d9aa" (UID: "f5abdaf7-4d6c-4473-b4b5-73246b90d9aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:02:20.337947 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.337845 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f5abdaf7-4d6c-4473-b4b5-73246b90d9aa" (UID: "f5abdaf7-4d6c-4473-b4b5-73246b90d9aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:02:20.337947 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.337849 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:02:20.337947 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.337884 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:02:20.338181 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.338152 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-data" (OuterVolumeSpecName: "istio-data") pod "f5abdaf7-4d6c-4473-b4b5-73246b90d9aa" (UID: "f5abdaf7-4d6c-4473-b4b5-73246b90d9aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:02:20.338246 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.338163 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f5abdaf7-4d6c-4473-b4b5-73246b90d9aa" (UID: "f5abdaf7-4d6c-4473-b4b5-73246b90d9aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:02:20.340432 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.340407 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f5abdaf7-4d6c-4473-b4b5-73246b90d9aa" (UID: "f5abdaf7-4d6c-4473-b4b5-73246b90d9aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 23:02:20.340566 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.340429 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-token" (OuterVolumeSpecName: "istio-token") pod "f5abdaf7-4d6c-4473-b4b5-73246b90d9aa" (UID: "f5abdaf7-4d6c-4473-b4b5-73246b90d9aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:02:20.340566 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.340427 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f5abdaf7-4d6c-4473-b4b5-73246b90d9aa" (UID: "f5abdaf7-4d6c-4473-b4b5-73246b90d9aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:02:20.340566 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.340521 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-kube-api-access-xmnn4" (OuterVolumeSpecName: "kube-api-access-xmnn4") pod "f5abdaf7-4d6c-4473-b4b5-73246b90d9aa" (UID: "f5abdaf7-4d6c-4473-b4b5-73246b90d9aa"). InnerVolumeSpecName "kube-api-access-xmnn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:02:20.355505 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.355459 2568 generic.go:356] "Generic (PLEG): container finished" podID="f5abdaf7-4d6c-4473-b4b5-73246b90d9aa" containerID="b4c950ca7504acea6c49a3ef716b8b0d8549f64a4e7294692cdcd2f4f47d82f7" exitCode=0 Jul 09 23:02:20.355674 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.355521 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" Jul 09 23:02:20.355674 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.355526 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" event={"ID":"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa","Type":"ContainerDied","Data":"b4c950ca7504acea6c49a3ef716b8b0d8549f64a4e7294692cdcd2f4f47d82f7"} Jul 09 23:02:20.355674 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.355596 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9" event={"ID":"f5abdaf7-4d6c-4473-b4b5-73246b90d9aa","Type":"ContainerDied","Data":"ae340fbfe15779cd5863f79dd97df4f7fb55fe693b3a22a418d44401b3e7da6f"} Jul 09 23:02:20.355674 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.355628 2568 scope.go:117] "RemoveContainer" containerID="b4c950ca7504acea6c49a3ef716b8b0d8549f64a4e7294692cdcd2f4f47d82f7" Jul 09 23:02:20.365089 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.365063 2568 scope.go:117] "RemoveContainer" containerID="b4c950ca7504acea6c49a3ef716b8b0d8549f64a4e7294692cdcd2f4f47d82f7" Jul 09 23:02:20.365398 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:02:20.365380 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c950ca7504acea6c49a3ef716b8b0d8549f64a4e7294692cdcd2f4f47d82f7\": container with ID starting with b4c950ca7504acea6c49a3ef716b8b0d8549f64a4e7294692cdcd2f4f47d82f7 not found: ID does not exist" containerID="b4c950ca7504acea6c49a3ef716b8b0d8549f64a4e7294692cdcd2f4f47d82f7" Jul 09 23:02:20.365468 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.365409 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c950ca7504acea6c49a3ef716b8b0d8549f64a4e7294692cdcd2f4f47d82f7"} err="failed to get container status \"b4c950ca7504acea6c49a3ef716b8b0d8549f64a4e7294692cdcd2f4f47d82f7\": rpc error: code = NotFound desc = could not find container \"b4c950ca7504acea6c49a3ef716b8b0d8549f64a4e7294692cdcd2f4f47d82f7\": container with ID starting with b4c950ca7504acea6c49a3ef716b8b0d8549f64a4e7294692cdcd2f4f47d82f7 not found: ID does not exist" Jul 09 23:02:20.377896 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.377861 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9"] Jul 09 23:02:20.388682 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.384031 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-202791fd-openshift-default-57c8f7cf59-zzlv9"] Jul 09 23:02:20.438563 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.438475 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:02:20.438563 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.438557 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:02:20.438563 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.438567 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmnn4\" (UniqueName: \"kubernetes.io/projected/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-kube-api-access-xmnn4\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:02:20.438780 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.438578 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:02:20.438780 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.438587 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:02:20.438780 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.438594 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:02:20.438780 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.438602 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:02:20.498825 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.498785 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:20.598619 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.598528 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:20.698484 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:20.698453 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:22.049654 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:22.049618 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5abdaf7-4d6c-4473-b4b5-73246b90d9aa" path="/var/lib/kubelet/pods/f5abdaf7-4d6c-4473-b4b5-73246b90d9aa/volumes" Jul 09 23:02:22.146741 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:22.146710 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:27.869324 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:27.869284 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:27.970679 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:27.970643 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:28.072655 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:28.072604 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:28.169349 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:28.169263 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:28.577826 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:28.577786 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:02:54.569451 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:02:54.569415 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:00.463371 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:00.463332 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:00.516035 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:00.515994 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:00.581971 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:00.581937 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:00.680385 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:00.680349 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:16.550757 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:16.550711 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:22.234662 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:22.234614 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:22.338186 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:22.338147 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:22.435276 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:22.435241 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:32.945106 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:32.945066 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:33.779754 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:33.779716 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:39.582354 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:39.582291 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:39.639707 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:39.639667 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:39.720985 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:39.720954 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:39.780161 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:39.780126 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:40.566847 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:40.566777 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:40.673799 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:40.673763 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:40.772444 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:40.772412 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:40.872140 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:40.872051 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:42.496122 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:42.496084 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:48.239596 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:48.239556 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:48.297692 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:48.297656 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:48.940719 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:48.940688 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:50.196963 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:50.196929 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:50.250938 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:50.250900 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:55.066561 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.066521 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:55.126828 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.126788 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:55.319284 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.319191 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:55.604160 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.604072 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr"] Jul 09 23:03:55.604438 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.604426 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a4b0e3e-f271-4a48-9e01-e7a562ced4c1" containerName="maas-api" Jul 09 23:03:55.604481 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.604439 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a4b0e3e-f271-4a48-9e01-e7a562ced4c1" containerName="maas-api" Jul 09 23:03:55.604481 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.604453 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5abdaf7-4d6c-4473-b4b5-73246b90d9aa" containerName="istio-proxy" Jul 09 23:03:55.604481 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.604459 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5abdaf7-4d6c-4473-b4b5-73246b90d9aa" containerName="istio-proxy" Jul 09 23:03:55.604609 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.604550 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5abdaf7-4d6c-4473-b4b5-73246b90d9aa" containerName="istio-proxy" Jul 09 23:03:55.604609 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.604559 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a4b0e3e-f271-4a48-9e01-e7a562ced4c1" containerName="maas-api" Jul 09 23:03:55.607148 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.607128 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.610266 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.610241 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2b5204df-openshift-default-dockercfg-vm467\"" Jul 09 23:03:55.625979 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.625939 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr"] Jul 09 23:03:55.627140 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.627107 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:55.710775 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.710720 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-podinfo\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.710775 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.710769 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03d7a55-c97c-47e9-8d2f-820f94483630-istiod-ca-cert\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.711048 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.710883 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjbr9\" (UniqueName: \"kubernetes.io/projected/a03d7a55-c97c-47e9-8d2f-820f94483630-kube-api-access-vjbr9\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.711048 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.710921 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-workload-certs\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.711048 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.710950 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-token\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.711207 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.711051 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-data\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.711207 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.711101 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-credential-socket\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.711207 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.711188 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-workload-socket\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.711358 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.711221 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-envoy\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.791056 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.791020 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:55.812293 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.812254 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-podinfo\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.812293 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.812294 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03d7a55-c97c-47e9-8d2f-820f94483630-istiod-ca-cert\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.812456 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.812381 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbr9\" (UniqueName: \"kubernetes.io/projected/a03d7a55-c97c-47e9-8d2f-820f94483630-kube-api-access-vjbr9\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.812456 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.812428 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-workload-certs\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.812456 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.812449 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-token\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.812716 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.812475 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-data\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.812716 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.812539 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-credential-socket\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.812716 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.812608 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-workload-socket\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.812875 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.812734 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-envoy\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.812875 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.812848 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-workload-certs\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.812989 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.812920 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-data\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.812989 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.812935 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-workload-socket\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.813100 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.813036 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-credential-socket\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.813168 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.813149 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03d7a55-c97c-47e9-8d2f-820f94483630-istiod-ca-cert\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.815113 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.815087 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-envoy\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.815256 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.815230 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-podinfo\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.820951 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.820926 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-token\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.821601 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.821578 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjbr9\" (UniqueName: \"kubernetes.io/projected/a03d7a55-c97c-47e9-8d2f-820f94483630-kube-api-access-vjbr9\") pod \"e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:55.921323 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.921222 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2b5204df-openshift-default-dockercfg-vm467\"" Jul 09 23:03:55.929595 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:55.929559 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:56.079594 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:56.079560 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr"] Jul 09 23:03:56.081922 ip-10-0-135-80 kubenswrapper[2568]: W0709 23:03:56.081880 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03d7a55_c97c_47e9_8d2f_820f94483630.slice/crio-7ffb5ef5b9f0218c97cda0dc7657290049411a74295ff19917d2f29fa4eec1c7 WatchSource:0}: Error finding container 7ffb5ef5b9f0218c97cda0dc7657290049411a74295ff19917d2f29fa4eec1c7: Status 404 returned error can't find the container with id 7ffb5ef5b9f0218c97cda0dc7657290049411a74295ff19917d2f29fa4eec1c7 Jul 09 23:03:56.084018 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:56.083987 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:03:56.084112 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:56.084057 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:03:56.084112 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:56.084087 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:03:56.860054 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:56.860013 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" event={"ID":"a03d7a55-c97c-47e9-8d2f-820f94483630","Type":"ContainerStarted","Data":"e9e2575f071797de83945cc9dd2ab4607b57e957d24340db4c82f7ac24aa1eae"} Jul 09 23:03:56.860249 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:56.860079 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" event={"ID":"a03d7a55-c97c-47e9-8d2f-820f94483630","Type":"ContainerStarted","Data":"7ffb5ef5b9f0218c97cda0dc7657290049411a74295ff19917d2f29fa4eec1c7"} Jul 09 23:03:56.881380 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:56.881317 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" podStartSLOduration=1.881298121 podStartE2EDuration="1.881298121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:03:56.879361733 +0000 UTC m=+1861.420440558" watchObservedRunningTime="2026-07-09 23:03:56.881298121 +0000 UTC m=+1861.422376990" Jul 09 23:03:56.930733 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:56.930688 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:57.934927 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:57.934898 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:58.830397 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:58.830359 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:58.870325 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:58.870296 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:58.871367 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:58.871347 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:03:58.914946 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:58.914902 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8"] Jul 09 23:03:58.918968 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:58.918938 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" Jul 09 23:03:58.919259 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:58.919196 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:58.921770 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:58.921751 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2b5204df-serving-cert\"" Jul 09 23:03:58.928463 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:58.928318 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8"] Jul 09 23:03:58.984911 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:58.984873 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:59.045454 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:59.045413 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01d8c11c-ed93-4f5e-a53e-de62f051aad4-maas-api-tls\") pod \"maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8\" (UID: \"01d8c11c-ed93-4f5e-a53e-de62f051aad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" Jul 09 23:03:59.045695 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:59.045510 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d8c11c-ed93-4f5e-a53e-de62f051aad4-openshift-service-ca\") pod \"maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8\" (UID: \"01d8c11c-ed93-4f5e-a53e-de62f051aad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" Jul 09 23:03:59.045695 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:59.045627 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx54g\" (UniqueName: \"kubernetes.io/projected/01d8c11c-ed93-4f5e-a53e-de62f051aad4-kube-api-access-hx54g\") pod \"maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8\" (UID: \"01d8c11c-ed93-4f5e-a53e-de62f051aad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" Jul 09 23:03:59.087230 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:59.087130 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:59.146623 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:59.146567 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx54g\" (UniqueName: \"kubernetes.io/projected/01d8c11c-ed93-4f5e-a53e-de62f051aad4-kube-api-access-hx54g\") pod \"maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8\" (UID: \"01d8c11c-ed93-4f5e-a53e-de62f051aad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" Jul 09 23:03:59.146623 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:59.146621 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01d8c11c-ed93-4f5e-a53e-de62f051aad4-maas-api-tls\") pod \"maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8\" (UID: \"01d8c11c-ed93-4f5e-a53e-de62f051aad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" Jul 09 23:03:59.146909 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:59.146692 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d8c11c-ed93-4f5e-a53e-de62f051aad4-openshift-service-ca\") pod \"maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8\" (UID: \"01d8c11c-ed93-4f5e-a53e-de62f051aad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" Jul 09 23:03:59.147413 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:59.147390 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d8c11c-ed93-4f5e-a53e-de62f051aad4-openshift-service-ca\") pod \"maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8\" (UID: \"01d8c11c-ed93-4f5e-a53e-de62f051aad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" Jul 09 23:03:59.149147 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:59.149121 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01d8c11c-ed93-4f5e-a53e-de62f051aad4-maas-api-tls\") pod \"maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8\" (UID: \"01d8c11c-ed93-4f5e-a53e-de62f051aad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" Jul 09 23:03:59.154905 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:59.154879 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx54g\" (UniqueName: \"kubernetes.io/projected/01d8c11c-ed93-4f5e-a53e-de62f051aad4-kube-api-access-hx54g\") pod \"maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8\" (UID: \"01d8c11c-ed93-4f5e-a53e-de62f051aad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" Jul 09 23:03:59.233760 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:59.233722 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" Jul 09 23:03:59.384594 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:59.384565 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8"] Jul 09 23:03:59.385805 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:59.385777 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:59.386548 ip-10-0-135-80 kubenswrapper[2568]: W0709 23:03:59.386527 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d8c11c_ed93_4f5e_a53e_de62f051aad4.slice/crio-50858f287ae8af221ddf4c1255a6e8bdffdbfc2394594a8cda8a16068e50da65 WatchSource:0}: Error finding container 50858f287ae8af221ddf4c1255a6e8bdffdbfc2394594a8cda8a16068e50da65: Status 404 returned error can't find the container with id 50858f287ae8af221ddf4c1255a6e8bdffdbfc2394594a8cda8a16068e50da65 Jul 09 23:03:59.486434 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:59.486400 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:03:59.881608 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:03:59.881569 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" event={"ID":"01d8c11c-ed93-4f5e-a53e-de62f051aad4","Type":"ContainerStarted","Data":"50858f287ae8af221ddf4c1255a6e8bdffdbfc2394594a8cda8a16068e50da65"} Jul 09 23:04:00.889245 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:00.889192 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" event={"ID":"01d8c11c-ed93-4f5e-a53e-de62f051aad4","Type":"ContainerStarted","Data":"d17afa6605c956b108800e9b30b64f88d4e6b33c17173b62708478c1a170db56"} Jul 09 23:04:00.889693 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:00.889479 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" Jul 09 23:04:00.906026 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:00.905973 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" podStartSLOduration=2.115410225 podStartE2EDuration="2.905955453s" totalImagesPullingTime="790.545228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:03:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:04:00.905158453 +0000 UTC m=+1865.446237249" watchObservedRunningTime="2026-07-09 23:04:00.905955453 +0000 UTC m=+1865.447034246" Jul 09 23:04:03.340736 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:03.340700 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:03.999641 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:03.999608 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:04.099461 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:04.099424 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:04.456077 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:04.456042 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:06.900450 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:06.900415 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" Jul 09 23:04:09.961871 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:09.961826 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:10.060991 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:10.060956 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:10.371373 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:10.371332 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:10.713106 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:10.713067 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8"] Jul 09 23:04:10.713362 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:10.713335 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" podUID="01d8c11c-ed93-4f5e-a53e-de62f051aad4" containerName="maas-api" containerID="cri-o://d17afa6605c956b108800e9b30b64f88d4e6b33c17173b62708478c1a170db56" gracePeriod=30 Jul 09 23:04:10.735700 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:10.735660 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:10.831624 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:10.831592 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:10.945453 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:10.945416 2568 generic.go:356] "Generic (PLEG): container finished" podID="01d8c11c-ed93-4f5e-a53e-de62f051aad4" containerID="d17afa6605c956b108800e9b30b64f88d4e6b33c17173b62708478c1a170db56" exitCode=0 Jul 09 23:04:10.945669 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:10.945514 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" event={"ID":"01d8c11c-ed93-4f5e-a53e-de62f051aad4","Type":"ContainerDied","Data":"d17afa6605c956b108800e9b30b64f88d4e6b33c17173b62708478c1a170db56"} Jul 09 23:04:10.991019 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:10.990981 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" Jul 09 23:04:11.161373 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:11.161327 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d8c11c-ed93-4f5e-a53e-de62f051aad4-openshift-service-ca\") pod \"01d8c11c-ed93-4f5e-a53e-de62f051aad4\" (UID: \"01d8c11c-ed93-4f5e-a53e-de62f051aad4\") " Jul 09 23:04:11.161603 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:11.161424 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx54g\" (UniqueName: \"kubernetes.io/projected/01d8c11c-ed93-4f5e-a53e-de62f051aad4-kube-api-access-hx54g\") pod \"01d8c11c-ed93-4f5e-a53e-de62f051aad4\" (UID: \"01d8c11c-ed93-4f5e-a53e-de62f051aad4\") " Jul 09 23:04:11.161603 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:11.161547 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01d8c11c-ed93-4f5e-a53e-de62f051aad4-maas-api-tls\") pod \"01d8c11c-ed93-4f5e-a53e-de62f051aad4\" (UID: \"01d8c11c-ed93-4f5e-a53e-de62f051aad4\") " Jul 09 23:04:11.161838 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:11.161810 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d8c11c-ed93-4f5e-a53e-de62f051aad4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "01d8c11c-ed93-4f5e-a53e-de62f051aad4" (UID: "01d8c11c-ed93-4f5e-a53e-de62f051aad4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:04:11.161943 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:11.161923 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d8c11c-ed93-4f5e-a53e-de62f051aad4-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:04:11.164193 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:11.164159 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01d8c11c-ed93-4f5e-a53e-de62f051aad4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "01d8c11c-ed93-4f5e-a53e-de62f051aad4" (UID: "01d8c11c-ed93-4f5e-a53e-de62f051aad4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:04:11.164315 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:11.164241 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d8c11c-ed93-4f5e-a53e-de62f051aad4-kube-api-access-hx54g" (OuterVolumeSpecName: "kube-api-access-hx54g") pod "01d8c11c-ed93-4f5e-a53e-de62f051aad4" (UID: "01d8c11c-ed93-4f5e-a53e-de62f051aad4"). InnerVolumeSpecName "kube-api-access-hx54g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:04:11.262523 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:11.262419 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx54g\" (UniqueName: \"kubernetes.io/projected/01d8c11c-ed93-4f5e-a53e-de62f051aad4-kube-api-access-hx54g\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:04:11.262523 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:11.262459 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01d8c11c-ed93-4f5e-a53e-de62f051aad4-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:04:11.951960 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:11.951918 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" event={"ID":"01d8c11c-ed93-4f5e-a53e-de62f051aad4","Type":"ContainerDied","Data":"50858f287ae8af221ddf4c1255a6e8bdffdbfc2394594a8cda8a16068e50da65"} Jul 09 23:04:11.951960 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:11.951961 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8" Jul 09 23:04:11.952222 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:11.952031 2568 scope.go:117] "RemoveContainer" containerID="d17afa6605c956b108800e9b30b64f88d4e6b33c17173b62708478c1a170db56" Jul 09 23:04:11.974519 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:11.974464 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8"] Jul 09 23:04:11.980383 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:11.980345 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b5204df-79c6d887b4-v2xt8"] Jul 09 23:04:12.048393 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:12.048356 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01d8c11c-ed93-4f5e-a53e-de62f051aad4" path="/var/lib/kubelet/pods/01d8c11c-ed93-4f5e-a53e-de62f051aad4/volumes" Jul 09 23:04:12.959299 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:12.959218 2568 generic.go:356] "Generic (PLEG): container finished" podID="a03d7a55-c97c-47e9-8d2f-820f94483630" containerID="e9e2575f071797de83945cc9dd2ab4607b57e957d24340db4c82f7ac24aa1eae" exitCode=0 Jul 09 23:04:12.959526 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:12.959297 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" event={"ID":"a03d7a55-c97c-47e9-8d2f-820f94483630","Type":"ContainerDied","Data":"e9e2575f071797de83945cc9dd2ab4607b57e957d24340db4c82f7ac24aa1eae"} Jul 09 23:04:12.959821 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:12.959797 2568 scope.go:117] "RemoveContainer" containerID="e9e2575f071797de83945cc9dd2ab4607b57e957d24340db4c82f7ac24aa1eae" Jul 09 23:04:12.960861 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:12.960837 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:04:12.960919 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:12.960904 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:04:12.960956 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:12.960936 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:04:13.871046 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:13.871008 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:04:13.968745 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:13.968712 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" event={"ID":"a03d7a55-c97c-47e9-8d2f-820f94483630","Type":"ContainerStarted","Data":"8b927e7a911c024bf303195563c8024f141cf0476256584a0310d3b9eec59fd7"} Jul 09 23:04:14.930462 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:14.930421 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:04:14.935178 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:14.935152 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:04:14.974709 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:14.974678 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:04:14.975576 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:14.975556 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:04:16.378103 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:16.378069 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:16.434885 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:16.434846 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:16.621779 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:16.621739 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr"] Jul 09 23:04:16.637356 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:16.637266 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:16.858169 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:16.858133 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:16.972773 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:16.972685 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:16.987017 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:16.986979 2568 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" secret="" err="secret \"e2e-mt-2b5204df-openshift-default-dockercfg-vm467\" not found" Jul 09 23:04:17.153658 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:17.153621 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:17.271144 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:17.271024 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:17.991710 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:17.991670 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" podUID="a03d7a55-c97c-47e9-8d2f-820f94483630" containerName="istio-proxy" containerID="cri-o://8b927e7a911c024bf303195563c8024f141cf0476256584a0310d3b9eec59fd7" gracePeriod=30 Jul 09 23:04:22.708652 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:22.708618 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:22.807150 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:22.807112 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:22.906271 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:22.906232 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:23.045238 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.045207 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:23.120231 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.120166 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:23.249172 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.249143 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:04:23.277465 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.277372 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-workload-certs\") pod \"a03d7a55-c97c-47e9-8d2f-820f94483630\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " Jul 09 23:04:23.277465 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.277418 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03d7a55-c97c-47e9-8d2f-820f94483630-istiod-ca-cert\") pod \"a03d7a55-c97c-47e9-8d2f-820f94483630\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " Jul 09 23:04:23.278102 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.278066 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a03d7a55-c97c-47e9-8d2f-820f94483630" (UID: "a03d7a55-c97c-47e9-8d2f-820f94483630"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:04:23.278382 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.278364 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a03d7a55-c97c-47e9-8d2f-820f94483630-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a03d7a55-c97c-47e9-8d2f-820f94483630" (UID: "a03d7a55-c97c-47e9-8d2f-820f94483630"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:04:23.378615 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.378580 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-podinfo\") pod \"a03d7a55-c97c-47e9-8d2f-820f94483630\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " Jul 09 23:04:23.378615 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.378620 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-token\") pod \"a03d7a55-c97c-47e9-8d2f-820f94483630\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " Jul 09 23:04:23.378883 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.378643 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-data\") pod \"a03d7a55-c97c-47e9-8d2f-820f94483630\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " Jul 09 23:04:23.378883 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.378729 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjbr9\" (UniqueName: \"kubernetes.io/projected/a03d7a55-c97c-47e9-8d2f-820f94483630-kube-api-access-vjbr9\") pod \"a03d7a55-c97c-47e9-8d2f-820f94483630\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " Jul 09 23:04:23.378883 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.378755 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-envoy\") pod \"a03d7a55-c97c-47e9-8d2f-820f94483630\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " Jul 09 23:04:23.378883 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.378784 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-credential-socket\") pod \"a03d7a55-c97c-47e9-8d2f-820f94483630\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " Jul 09 23:04:23.378883 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.378866 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-workload-socket\") pod \"a03d7a55-c97c-47e9-8d2f-820f94483630\" (UID: \"a03d7a55-c97c-47e9-8d2f-820f94483630\") " Jul 09 23:04:23.379155 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.379123 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:04:23.379155 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.379142 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a03d7a55-c97c-47e9-8d2f-820f94483630-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:04:23.379257 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.379166 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a03d7a55-c97c-47e9-8d2f-820f94483630" (UID: "a03d7a55-c97c-47e9-8d2f-820f94483630"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:04:23.379341 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.379321 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a03d7a55-c97c-47e9-8d2f-820f94483630" (UID: "a03d7a55-c97c-47e9-8d2f-820f94483630"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:04:23.379654 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.379630 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-data" (OuterVolumeSpecName: "istio-data") pod "a03d7a55-c97c-47e9-8d2f-820f94483630" (UID: "a03d7a55-c97c-47e9-8d2f-820f94483630"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:04:23.380926 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.380888 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03d7a55-c97c-47e9-8d2f-820f94483630-kube-api-access-vjbr9" (OuterVolumeSpecName: "kube-api-access-vjbr9") pod "a03d7a55-c97c-47e9-8d2f-820f94483630" (UID: "a03d7a55-c97c-47e9-8d2f-820f94483630"). InnerVolumeSpecName "kube-api-access-vjbr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:04:23.381056 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.380929 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-token" (OuterVolumeSpecName: "istio-token") pod "a03d7a55-c97c-47e9-8d2f-820f94483630" (UID: "a03d7a55-c97c-47e9-8d2f-820f94483630"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:04:23.381227 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.381211 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a03d7a55-c97c-47e9-8d2f-820f94483630" (UID: "a03d7a55-c97c-47e9-8d2f-820f94483630"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:04:23.381300 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.381224 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a03d7a55-c97c-47e9-8d2f-820f94483630" (UID: "a03d7a55-c97c-47e9-8d2f-820f94483630"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 23:04:23.480212 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.480172 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjbr9\" (UniqueName: \"kubernetes.io/projected/a03d7a55-c97c-47e9-8d2f-820f94483630-kube-api-access-vjbr9\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:04:23.480212 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.480203 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:04:23.480212 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.480213 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:04:23.480212 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.480221 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:04:23.480531 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.480229 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:04:23.480531 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.480237 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:04:23.480531 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:23.480245 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a03d7a55-c97c-47e9-8d2f-820f94483630-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:04:24.025317 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:24.025278 2568 generic.go:356] "Generic (PLEG): container finished" podID="a03d7a55-c97c-47e9-8d2f-820f94483630" containerID="8b927e7a911c024bf303195563c8024f141cf0476256584a0310d3b9eec59fd7" exitCode=0 Jul 09 23:04:24.025854 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:24.025357 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" Jul 09 23:04:24.025854 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:24.025356 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" event={"ID":"a03d7a55-c97c-47e9-8d2f-820f94483630","Type":"ContainerDied","Data":"8b927e7a911c024bf303195563c8024f141cf0476256584a0310d3b9eec59fd7"} Jul 09 23:04:24.025854 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:24.025529 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr" event={"ID":"a03d7a55-c97c-47e9-8d2f-820f94483630","Type":"ContainerDied","Data":"7ffb5ef5b9f0218c97cda0dc7657290049411a74295ff19917d2f29fa4eec1c7"} Jul 09 23:04:24.025854 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:24.025555 2568 scope.go:117] "RemoveContainer" containerID="8b927e7a911c024bf303195563c8024f141cf0476256584a0310d3b9eec59fd7" Jul 09 23:04:24.035728 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:24.035708 2568 scope.go:117] "RemoveContainer" containerID="e9e2575f071797de83945cc9dd2ab4607b57e957d24340db4c82f7ac24aa1eae" Jul 09 23:04:24.044692 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:24.044671 2568 scope.go:117] "RemoveContainer" containerID="8b927e7a911c024bf303195563c8024f141cf0476256584a0310d3b9eec59fd7" Jul 09 23:04:24.044998 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:04:24.044976 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b927e7a911c024bf303195563c8024f141cf0476256584a0310d3b9eec59fd7\": container with ID starting with 8b927e7a911c024bf303195563c8024f141cf0476256584a0310d3b9eec59fd7 not found: ID does not exist" containerID="8b927e7a911c024bf303195563c8024f141cf0476256584a0310d3b9eec59fd7" Jul 09 23:04:24.045055 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:24.045008 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b927e7a911c024bf303195563c8024f141cf0476256584a0310d3b9eec59fd7"} err="failed to get container status \"8b927e7a911c024bf303195563c8024f141cf0476256584a0310d3b9eec59fd7\": rpc error: code = NotFound desc = could not find container \"8b927e7a911c024bf303195563c8024f141cf0476256584a0310d3b9eec59fd7\": container with ID starting with 8b927e7a911c024bf303195563c8024f141cf0476256584a0310d3b9eec59fd7 not found: ID does not exist" Jul 09 23:04:24.045055 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:24.045027 2568 scope.go:117] "RemoveContainer" containerID="e9e2575f071797de83945cc9dd2ab4607b57e957d24340db4c82f7ac24aa1eae" Jul 09 23:04:24.045296 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:04:24.045279 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e2575f071797de83945cc9dd2ab4607b57e957d24340db4c82f7ac24aa1eae\": container with ID starting with e9e2575f071797de83945cc9dd2ab4607b57e957d24340db4c82f7ac24aa1eae not found: ID does not exist" containerID="e9e2575f071797de83945cc9dd2ab4607b57e957d24340db4c82f7ac24aa1eae" Jul 09 23:04:24.045354 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:24.045299 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e2575f071797de83945cc9dd2ab4607b57e957d24340db4c82f7ac24aa1eae"} err="failed to get container status \"e9e2575f071797de83945cc9dd2ab4607b57e957d24340db4c82f7ac24aa1eae\": rpc error: code = NotFound desc = could not find container \"e9e2575f071797de83945cc9dd2ab4607b57e957d24340db4c82f7ac24aa1eae\": container with ID starting with e9e2575f071797de83945cc9dd2ab4607b57e957d24340db4c82f7ac24aa1eae not found: ID does not exist" Jul 09 23:04:24.049698 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:24.049668 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr"] Jul 09 23:04:24.053330 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:24.053300 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2b5204df-openshift-default-df8bb9b7-nmgpr"] Jul 09 23:04:26.047966 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:26.047930 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a03d7a55-c97c-47e9-8d2f-820f94483630" path="/var/lib/kubelet/pods/a03d7a55-c97c-47e9-8d2f-820f94483630/volumes" Jul 09 23:04:28.637548 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:28.637480 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:28.697059 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:28.697021 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:35.490706 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:35.490665 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:35.830200 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:35.830163 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:35.929205 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:35.929171 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:38.460258 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.460218 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:38.561272 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.561234 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:38.629359 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.629318 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79"] Jul 09 23:04:38.629768 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.629749 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a03d7a55-c97c-47e9-8d2f-820f94483630" containerName="istio-proxy" Jul 09 23:04:38.629858 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.629770 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a03d7a55-c97c-47e9-8d2f-820f94483630" containerName="istio-proxy" Jul 09 23:04:38.629858 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.629809 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01d8c11c-ed93-4f5e-a53e-de62f051aad4" containerName="maas-api" Jul 09 23:04:38.629858 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.629818 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="01d8c11c-ed93-4f5e-a53e-de62f051aad4" containerName="maas-api" Jul 09 23:04:38.630013 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.629908 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a03d7a55-c97c-47e9-8d2f-820f94483630" containerName="istio-proxy" Jul 09 23:04:38.630013 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.629922 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a03d7a55-c97c-47e9-8d2f-820f94483630" containerName="istio-proxy" Jul 09 23:04:38.630013 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.629934 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="01d8c11c-ed93-4f5e-a53e-de62f051aad4" containerName="maas-api" Jul 09 23:04:38.632091 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.632067 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" Jul 09 23:04:38.634876 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.634849 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4e2b27a4-serving-cert\"" Jul 09 23:04:38.644233 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.644202 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79"] Jul 09 23:04:38.672276 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.672233 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:38.726195 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.726098 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5af749e-d767-4516-b334-ebcfa39b06d4-openshift-service-ca\") pod \"maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79\" (UID: \"e5af749e-d767-4516-b334-ebcfa39b06d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" Jul 09 23:04:38.726365 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.726203 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5af749e-d767-4516-b334-ebcfa39b06d4-maas-api-tls\") pod \"maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79\" (UID: \"e5af749e-d767-4516-b334-ebcfa39b06d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" Jul 09 23:04:38.726365 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.726253 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5b7b\" (UniqueName: \"kubernetes.io/projected/e5af749e-d767-4516-b334-ebcfa39b06d4-kube-api-access-v5b7b\") pod \"maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79\" (UID: \"e5af749e-d767-4516-b334-ebcfa39b06d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" Jul 09 23:04:38.769966 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.769916 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:38.826722 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.826684 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5af749e-d767-4516-b334-ebcfa39b06d4-openshift-service-ca\") pod \"maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79\" (UID: \"e5af749e-d767-4516-b334-ebcfa39b06d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" Jul 09 23:04:38.826923 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.826767 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5af749e-d767-4516-b334-ebcfa39b06d4-maas-api-tls\") pod \"maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79\" (UID: \"e5af749e-d767-4516-b334-ebcfa39b06d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" Jul 09 23:04:38.826923 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.826797 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5b7b\" (UniqueName: \"kubernetes.io/projected/e5af749e-d767-4516-b334-ebcfa39b06d4-kube-api-access-v5b7b\") pod \"maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79\" (UID: \"e5af749e-d767-4516-b334-ebcfa39b06d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" Jul 09 23:04:38.827354 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.827323 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5af749e-d767-4516-b334-ebcfa39b06d4-openshift-service-ca\") pod \"maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79\" (UID: \"e5af749e-d767-4516-b334-ebcfa39b06d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" Jul 09 23:04:38.829253 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.829227 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5af749e-d767-4516-b334-ebcfa39b06d4-maas-api-tls\") pod \"maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79\" (UID: \"e5af749e-d767-4516-b334-ebcfa39b06d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" Jul 09 23:04:38.837063 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.837006 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5b7b\" (UniqueName: \"kubernetes.io/projected/e5af749e-d767-4516-b334-ebcfa39b06d4-kube-api-access-v5b7b\") pod \"maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79\" (UID: \"e5af749e-d767-4516-b334-ebcfa39b06d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" Jul 09 23:04:38.944185 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:38.944145 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" Jul 09 23:04:39.109232 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:39.109199 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79"] Jul 09 23:04:39.110555 ip-10-0-135-80 kubenswrapper[2568]: W0709 23:04:39.110530 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5af749e_d767_4516_b334_ebcfa39b06d4.slice/crio-adfa0e74299e9a71b67fa0dda5d33b3a1302023c00b971aa43b509b59f10a48d WatchSource:0}: Error finding container adfa0e74299e9a71b67fa0dda5d33b3a1302023c00b971aa43b509b59f10a48d: Status 404 returned error can't find the container with id adfa0e74299e9a71b67fa0dda5d33b3a1302023c00b971aa43b509b59f10a48d Jul 09 23:04:39.310832 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:39.310796 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:39.408911 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:39.408869 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:40.112664 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:40.112630 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" event={"ID":"e5af749e-d767-4516-b334-ebcfa39b06d4","Type":"ContainerStarted","Data":"15cae46a65ccd13730bf40d963142e2a75633f3b4e7983b11f8c3ac5495447c8"} Jul 09 23:04:40.113104 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:40.112680 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" event={"ID":"e5af749e-d767-4516-b334-ebcfa39b06d4","Type":"ContainerStarted","Data":"adfa0e74299e9a71b67fa0dda5d33b3a1302023c00b971aa43b509b59f10a48d"} Jul 09 23:04:40.113104 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:40.112775 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" Jul 09 23:04:40.129001 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:40.128948 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" podStartSLOduration=1.775243171 podStartE2EDuration="2.128928339s" totalImagesPullingTime="353.685168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:04:40.127804842 +0000 UTC m=+1904.668883637" watchObservedRunningTime="2026-07-09 23:04:40.128928339 +0000 UTC m=+1904.670007132" Jul 09 23:04:42.979909 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:42.979872 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:46.123386 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:46.123357 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" Jul 09 23:04:51.077015 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:51.076974 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:04:51.140778 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:04:51.140742 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:05:00.687813 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:00.687760 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:05:00.788327 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:00.788287 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:05:10.279763 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:10.279723 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:05:10.617411 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:10.617374 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79"] Jul 09 23:05:10.617693 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:10.617647 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" podUID="e5af749e-d767-4516-b334-ebcfa39b06d4" containerName="maas-api" containerID="cri-o://15cae46a65ccd13730bf40d963142e2a75633f3b4e7983b11f8c3ac5495447c8" gracePeriod=30 Jul 09 23:05:10.642623 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:10.642592 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:05:10.868717 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:10.868644 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" Jul 09 23:05:10.911840 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:10.911805 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5af749e-d767-4516-b334-ebcfa39b06d4-maas-api-tls\") pod \"e5af749e-d767-4516-b334-ebcfa39b06d4\" (UID: \"e5af749e-d767-4516-b334-ebcfa39b06d4\") " Jul 09 23:05:10.912025 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:10.911851 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5b7b\" (UniqueName: \"kubernetes.io/projected/e5af749e-d767-4516-b334-ebcfa39b06d4-kube-api-access-v5b7b\") pod \"e5af749e-d767-4516-b334-ebcfa39b06d4\" (UID: \"e5af749e-d767-4516-b334-ebcfa39b06d4\") " Jul 09 23:05:10.912025 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:10.911999 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5af749e-d767-4516-b334-ebcfa39b06d4-openshift-service-ca\") pod \"e5af749e-d767-4516-b334-ebcfa39b06d4\" (UID: \"e5af749e-d767-4516-b334-ebcfa39b06d4\") " Jul 09 23:05:10.912368 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:10.912345 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5af749e-d767-4516-b334-ebcfa39b06d4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e5af749e-d767-4516-b334-ebcfa39b06d4" (UID: "e5af749e-d767-4516-b334-ebcfa39b06d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:05:10.914098 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:10.914070 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5af749e-d767-4516-b334-ebcfa39b06d4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e5af749e-d767-4516-b334-ebcfa39b06d4" (UID: "e5af749e-d767-4516-b334-ebcfa39b06d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:05:10.914098 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:10.914081 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5af749e-d767-4516-b334-ebcfa39b06d4-kube-api-access-v5b7b" (OuterVolumeSpecName: "kube-api-access-v5b7b") pod "e5af749e-d767-4516-b334-ebcfa39b06d4" (UID: "e5af749e-d767-4516-b334-ebcfa39b06d4"). InnerVolumeSpecName "kube-api-access-v5b7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:05:11.012824 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:11.012786 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5af749e-d767-4516-b334-ebcfa39b06d4-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:05:11.012824 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:11.012820 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5af749e-d767-4516-b334-ebcfa39b06d4-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:05:11.012824 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:11.012829 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5b7b\" (UniqueName: \"kubernetes.io/projected/e5af749e-d767-4516-b334-ebcfa39b06d4-kube-api-access-v5b7b\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:05:11.278826 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:11.278729 2568 generic.go:356] "Generic (PLEG): container finished" podID="e5af749e-d767-4516-b334-ebcfa39b06d4" containerID="15cae46a65ccd13730bf40d963142e2a75633f3b4e7983b11f8c3ac5495447c8" exitCode=0 Jul 09 23:05:11.278826 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:11.278778 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" Jul 09 23:05:11.279020 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:11.278820 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" event={"ID":"e5af749e-d767-4516-b334-ebcfa39b06d4","Type":"ContainerDied","Data":"15cae46a65ccd13730bf40d963142e2a75633f3b4e7983b11f8c3ac5495447c8"} Jul 09 23:05:11.279020 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:11.278876 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79" event={"ID":"e5af749e-d767-4516-b334-ebcfa39b06d4","Type":"ContainerDied","Data":"adfa0e74299e9a71b67fa0dda5d33b3a1302023c00b971aa43b509b59f10a48d"} Jul 09 23:05:11.279020 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:11.278900 2568 scope.go:117] "RemoveContainer" containerID="15cae46a65ccd13730bf40d963142e2a75633f3b4e7983b11f8c3ac5495447c8" Jul 09 23:05:11.288623 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:11.288600 2568 scope.go:117] "RemoveContainer" containerID="15cae46a65ccd13730bf40d963142e2a75633f3b4e7983b11f8c3ac5495447c8" Jul 09 23:05:11.288921 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:05:11.288910 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15cae46a65ccd13730bf40d963142e2a75633f3b4e7983b11f8c3ac5495447c8\": container with ID starting with 15cae46a65ccd13730bf40d963142e2a75633f3b4e7983b11f8c3ac5495447c8 not found: ID does not exist" containerID="15cae46a65ccd13730bf40d963142e2a75633f3b4e7983b11f8c3ac5495447c8" Jul 09 23:05:11.288965 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:11.288934 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15cae46a65ccd13730bf40d963142e2a75633f3b4e7983b11f8c3ac5495447c8"} err="failed to get container status \"15cae46a65ccd13730bf40d963142e2a75633f3b4e7983b11f8c3ac5495447c8\": rpc error: code = NotFound desc = could not find container \"15cae46a65ccd13730bf40d963142e2a75633f3b4e7983b11f8c3ac5495447c8\": container with ID starting with 15cae46a65ccd13730bf40d963142e2a75633f3b4e7983b11f8c3ac5495447c8 not found: ID does not exist" Jul 09 23:05:11.307389 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:11.307348 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79"] Jul 09 23:05:11.307620 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:11.307417 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e2b27a4-69889c4b8d-rpt79"] Jul 09 23:05:12.048614 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:12.048567 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5af749e-d767-4516-b334-ebcfa39b06d4" path="/var/lib/kubelet/pods/e5af749e-d767-4516-b334-ebcfa39b06d4/volumes" Jul 09 23:05:16.424399 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:16.424363 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:05:21.499134 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:21.499096 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:05:58.075731 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:58.075695 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:05:59.197473 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:05:59.197429 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:03.746718 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:03.746682 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:03.811416 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:03.811369 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:04.131762 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:04.131718 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:04.274707 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:04.274674 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:05.358298 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:05.358250 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:05.471296 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:05.471263 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:05.581309 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:05.581264 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:06.336554 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:06.336516 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:06.436520 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:06.436450 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:06.537641 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:06.537608 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:07.954897 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:07.954858 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:13.553310 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:13.553273 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:13.629956 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:13.629923 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:14.233539 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:14.233502 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:20.159473 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:20.159430 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:20.217710 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:20.217670 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:20.292802 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:20.292769 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:20.391512 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:20.391458 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:30.894998 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:30.894956 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:31.230924 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:31.230799 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:52.211430 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:52.211390 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:57.949393 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:57.949357 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:58.049462 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.049410 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:58.311883 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.311847 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:58.602904 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.602817 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx"] Jul 09 23:06:58.603178 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.603166 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5af749e-d767-4516-b334-ebcfa39b06d4" containerName="maas-api" Jul 09 23:06:58.603221 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.603179 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5af749e-d767-4516-b334-ebcfa39b06d4" containerName="maas-api" Jul 09 23:06:58.603221 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.603195 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a03d7a55-c97c-47e9-8d2f-820f94483630" containerName="istio-proxy" Jul 09 23:06:58.603221 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.603201 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a03d7a55-c97c-47e9-8d2f-820f94483630" containerName="istio-proxy" Jul 09 23:06:58.603319 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.603260 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5af749e-d767-4516-b334-ebcfa39b06d4" containerName="maas-api" Jul 09 23:06:58.606246 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.606226 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.609144 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.609121 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ed9500-openshift-default-dockercfg-zzv89\"" Jul 09 23:06:58.619139 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.619109 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx"] Jul 09 23:06:58.622842 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.622794 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:58.666081 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.666046 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-workload-socket\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.666266 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.666104 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-istio-data\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.666266 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.666176 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29eeb17-f328-4403-a427-c0c7b8047349-istio-podinfo\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.666266 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.666220 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29eeb17-f328-4403-a427-c0c7b8047349-istio-token\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.666266 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.666244 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-istio-envoy\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.666416 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.666288 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29eeb17-f328-4403-a427-c0c7b8047349-istiod-ca-cert\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.666416 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.666317 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-credential-socket\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.666416 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.666374 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-workload-certs\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.666538 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.666416 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fkpk\" (UniqueName: \"kubernetes.io/projected/e29eeb17-f328-4403-a427-c0c7b8047349-kube-api-access-4fkpk\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.767618 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.767571 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-istio-data\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.767837 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.767629 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29eeb17-f328-4403-a427-c0c7b8047349-istio-podinfo\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.767837 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.767668 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29eeb17-f328-4403-a427-c0c7b8047349-istio-token\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.767837 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.767699 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-istio-envoy\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.768021 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.767850 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29eeb17-f328-4403-a427-c0c7b8047349-istiod-ca-cert\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.768021 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.767898 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-credential-socket\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.768021 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.767931 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-workload-certs\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.768021 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.767955 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fkpk\" (UniqueName: \"kubernetes.io/projected/e29eeb17-f328-4403-a427-c0c7b8047349-kube-api-access-4fkpk\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.768021 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.768002 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-istio-data\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.768021 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.768010 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-workload-socket\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.768276 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.768229 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-workload-certs\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.768332 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.768300 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-credential-socket\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.768399 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.768376 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-workload-socket\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.768538 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.768516 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29eeb17-f328-4403-a427-c0c7b8047349-istiod-ca-cert\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.770074 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.770051 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-istio-envoy\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.770400 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.770383 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29eeb17-f328-4403-a427-c0c7b8047349-istio-podinfo\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.775826 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.775799 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29eeb17-f328-4403-a427-c0c7b8047349-istio-token\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.775942 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.775874 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fkpk\" (UniqueName: \"kubernetes.io/projected/e29eeb17-f328-4403-a427-c0c7b8047349-kube-api-access-4fkpk\") pod \"e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:58.850392 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.850340 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:06:58.918510 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:58.918387 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:06:59.064242 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:59.064185 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx"] Jul 09 23:06:59.066670 ip-10-0-135-80 kubenswrapper[2568]: W0709 23:06:59.066639 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29eeb17_f328_4403_a427_c0c7b8047349.slice/crio-e8a2b735e77f42be56bb2e17d8c747e926587afbb62c536a42211e6a8936efb7 WatchSource:0}: Error finding container e8a2b735e77f42be56bb2e17d8c747e926587afbb62c536a42211e6a8936efb7: Status 404 returned error can't find the container with id e8a2b735e77f42be56bb2e17d8c747e926587afbb62c536a42211e6a8936efb7 Jul 09 23:06:59.068862 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:59.068828 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:06:59.068956 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:59.068899 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:06:59.068956 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:59.068931 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:06:59.863461 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:59.863416 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" event={"ID":"e29eeb17-f328-4403-a427-c0c7b8047349","Type":"ContainerStarted","Data":"7d8bc4161e391e1ccf60f3b079c68daaae5941b068616468a5c43a7a28e2f1f2"} Jul 09 23:06:59.863687 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:59.863519 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" event={"ID":"e29eeb17-f328-4403-a427-c0c7b8047349","Type":"ContainerStarted","Data":"e8a2b735e77f42be56bb2e17d8c747e926587afbb62c536a42211e6a8936efb7"} Jul 09 23:06:59.887052 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:59.886979 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" podStartSLOduration=1.88695433 podStartE2EDuration="1.88695433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:06:59.886706784 +0000 UTC m=+2044.427785579" watchObservedRunningTime="2026-07-09 23:06:59.88695433 +0000 UTC m=+2044.428033124" Jul 09 23:06:59.918946 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:06:59.918909 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:07:00.923930 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:00.923898 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:07:01.854222 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:01.854189 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:01.875187 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:01.875150 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:07:01.876173 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:01.876145 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:07:01.953694 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:01.953662 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:02.039550 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:02.039513 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4"] Jul 09 23:07:02.042808 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:02.042781 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" Jul 09 23:07:02.047291 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:02.047258 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ed9500-serving-cert\"" Jul 09 23:07:02.059268 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:02.059236 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4"] Jul 09 23:07:02.100196 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:02.100155 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:02.100422 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:02.100404 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e97cc60-4bba-4804-99a3-39edeb824436-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4\" (UID: \"2e97cc60-4bba-4804-99a3-39edeb824436\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" Jul 09 23:07:02.100471 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:02.100457 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e97cc60-4bba-4804-99a3-39edeb824436-maas-api-tls\") pod \"maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4\" (UID: \"2e97cc60-4bba-4804-99a3-39edeb824436\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" Jul 09 23:07:02.100612 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:02.100587 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p4rr\" (UniqueName: \"kubernetes.io/projected/2e97cc60-4bba-4804-99a3-39edeb824436-kube-api-access-5p4rr\") pod \"maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4\" (UID: \"2e97cc60-4bba-4804-99a3-39edeb824436\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" Jul 09 23:07:02.202253 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:02.202151 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e97cc60-4bba-4804-99a3-39edeb824436-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4\" (UID: \"2e97cc60-4bba-4804-99a3-39edeb824436\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" Jul 09 23:07:02.202253 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:02.202202 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e97cc60-4bba-4804-99a3-39edeb824436-maas-api-tls\") pod \"maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4\" (UID: \"2e97cc60-4bba-4804-99a3-39edeb824436\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" Jul 09 23:07:02.202253 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:02.202238 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p4rr\" (UniqueName: \"kubernetes.io/projected/2e97cc60-4bba-4804-99a3-39edeb824436-kube-api-access-5p4rr\") pod \"maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4\" (UID: \"2e97cc60-4bba-4804-99a3-39edeb824436\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" Jul 09 23:07:02.202595 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:07:02.202376 2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-serving-cert: secret "maas-api-e2e-inf-a-ed9500-serving-cert" not found Jul 09 23:07:02.202595 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:07:02.202446 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e97cc60-4bba-4804-99a3-39edeb824436-maas-api-tls podName:2e97cc60-4bba-4804-99a3-39edeb824436 nodeName:}" failed. No retries permitted until 2026-07-09 23:07:02.702423498 +0000 UTC m=+2047.243502269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2e97cc60-4bba-4804-99a3-39edeb824436-maas-api-tls") pod "maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" (UID: "2e97cc60-4bba-4804-99a3-39edeb824436") : secret "maas-api-e2e-inf-a-ed9500-serving-cert" not found Jul 09 23:07:02.202990 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:02.202958 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e97cc60-4bba-4804-99a3-39edeb824436-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4\" (UID: \"2e97cc60-4bba-4804-99a3-39edeb824436\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" Jul 09 23:07:02.218163 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:02.218124 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p4rr\" (UniqueName: \"kubernetes.io/projected/2e97cc60-4bba-4804-99a3-39edeb824436-kube-api-access-5p4rr\") pod \"maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4\" (UID: \"2e97cc60-4bba-4804-99a3-39edeb824436\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" Jul 09 23:07:02.362778 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:02.362731 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:02.545324 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:02.545223 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:02.646249 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:02.646198 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:02.707809 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:02.707774 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e97cc60-4bba-4804-99a3-39edeb824436-maas-api-tls\") pod \"maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4\" (UID: \"2e97cc60-4bba-4804-99a3-39edeb824436\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" Jul 09 23:07:02.710479 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:02.710445 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e97cc60-4bba-4804-99a3-39edeb824436-maas-api-tls\") pod \"maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4\" (UID: \"2e97cc60-4bba-4804-99a3-39edeb824436\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" Jul 09 23:07:02.954164 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:02.954125 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" Jul 09 23:07:03.098285 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:03.098259 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4"] Jul 09 23:07:03.100176 ip-10-0-135-80 kubenswrapper[2568]: W0709 23:07:03.100143 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e97cc60_4bba_4804_99a3_39edeb824436.slice/crio-3072ef61727f0717ceba638c1089d46a3d3cdeeae42303acda995f0310ee1c9b WatchSource:0}: Error finding container 3072ef61727f0717ceba638c1089d46a3d3cdeeae42303acda995f0310ee1c9b: Status 404 returned error can't find the container with id 3072ef61727f0717ceba638c1089d46a3d3cdeeae42303acda995f0310ee1c9b Jul 09 23:07:03.101897 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:03.101873 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 23:07:03.886936 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:03.886877 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" event={"ID":"2e97cc60-4bba-4804-99a3-39edeb824436","Type":"ContainerStarted","Data":"10427c26cb0b23eed2001071a861a146a22a846d0ec278fa8e0baf152c543f87"} Jul 09 23:07:03.887322 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:03.886951 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" event={"ID":"2e97cc60-4bba-4804-99a3-39edeb824436","Type":"ContainerStarted","Data":"3072ef61727f0717ceba638c1089d46a3d3cdeeae42303acda995f0310ee1c9b"} Jul 09 23:07:03.887322 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:03.887153 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" Jul 09 23:07:03.904749 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:03.904694 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" podStartSLOduration=2.239198124 podStartE2EDuration="2.904675191s" totalImagesPullingTime="665.477067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:07:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:07:03.903102116 +0000 UTC m=+2048.444180913" watchObservedRunningTime="2026-07-09 23:07:03.904675191 +0000 UTC m=+2048.445753985" Jul 09 23:07:06.381045 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:06.381003 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:06.726237 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:06.726124 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:07.039126 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:07.039039 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:07.149145 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:07.149112 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:09.897731 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:09.897698 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" Jul 09 23:07:10.226860 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.226829 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:10.318844 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.318802 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:10.326183 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.326143 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl"] Jul 09 23:07:10.330443 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.330415 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" Jul 09 23:07:10.335543 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.335511 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ceba72-serving-cert\"" Jul 09 23:07:10.339666 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.339148 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl"] Jul 09 23:07:10.483179 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.483054 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b-maas-api-tls\") pod \"maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl\" (UID: \"5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" Jul 09 23:07:10.483385 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.483246 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl\" (UID: \"5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" Jul 09 23:07:10.483385 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.483302 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29jxt\" (UniqueName: \"kubernetes.io/projected/5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b-kube-api-access-29jxt\") pod \"maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl\" (UID: \"5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" Jul 09 23:07:10.580415 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.580351 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:10.584432 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.584379 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl\" (UID: \"5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" Jul 09 23:07:10.584720 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.584691 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29jxt\" (UniqueName: \"kubernetes.io/projected/5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b-kube-api-access-29jxt\") pod \"maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl\" (UID: \"5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" Jul 09 23:07:10.585020 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.584985 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b-maas-api-tls\") pod \"maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl\" (UID: \"5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" Jul 09 23:07:10.586378 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.586351 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl\" (UID: \"5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" Jul 09 23:07:10.588599 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.588570 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b-maas-api-tls\") pod \"maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl\" (UID: \"5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" Jul 09 23:07:10.605135 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.605095 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29jxt\" (UniqueName: \"kubernetes.io/projected/5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b-kube-api-access-29jxt\") pod \"maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl\" (UID: \"5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" Jul 09 23:07:10.644182 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.644139 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" Jul 09 23:07:10.687080 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.686649 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:10.830200 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.830169 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl"] Jul 09 23:07:10.831609 ip-10-0-135-80 kubenswrapper[2568]: W0709 23:07:10.831584 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5ee1ba_3dd8_4ac3_a2ae_f1985f54985b.slice/crio-ed35b18cfe1a87a2815bf7779e76363b75aacf899a7f6bf6792ba2afc80c2545 WatchSource:0}: Error finding container ed35b18cfe1a87a2815bf7779e76363b75aacf899a7f6bf6792ba2afc80c2545: Status 404 returned error can't find the container with id ed35b18cfe1a87a2815bf7779e76363b75aacf899a7f6bf6792ba2afc80c2545 Jul 09 23:07:10.928469 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:10.928438 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" event={"ID":"5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b","Type":"ContainerStarted","Data":"ed35b18cfe1a87a2815bf7779e76363b75aacf899a7f6bf6792ba2afc80c2545"} Jul 09 23:07:11.028210 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:11.028109 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:11.128805 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:11.128773 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:11.935659 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:11.935620 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" event={"ID":"5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b","Type":"ContainerStarted","Data":"a4a4cd817625d6681e8824dd6c9d08d25de7f3947cae59d2c6694b167da9eb2f"} Jul 09 23:07:11.936064 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:11.935789 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" Jul 09 23:07:11.951784 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:11.951732 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" podStartSLOduration=1.466834624 podStartE2EDuration="1.951713329s" totalImagesPullingTime="484.878705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:07:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:07:11.950901013 +0000 UTC m=+2056.491979809" watchObservedRunningTime="2026-07-09 23:07:11.951713329 +0000 UTC m=+2056.492792122" Jul 09 23:07:14.755075 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:14.755039 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:17.318054 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:17.317170 2568 patch_prober.go:28] interesting pod/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:50534->10.134.0.44:15021: read: connection reset by peer" start-of-body= Jul 09 23:07:17.318054 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:17.317307 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" podUID="e29eeb17-f328-4403-a427-c0c7b8047349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:50534->10.134.0.44:15021: read: connection reset by peer" Jul 09 23:07:17.947152 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:17.947125 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" Jul 09 23:07:17.970746 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:17.970698 2568 generic.go:356] "Generic (PLEG): container finished" podID="e29eeb17-f328-4403-a427-c0c7b8047349" containerID="7d8bc4161e391e1ccf60f3b079c68daaae5941b068616468a5c43a7a28e2f1f2" exitCode=0 Jul 09 23:07:17.970746 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:17.970736 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" event={"ID":"e29eeb17-f328-4403-a427-c0c7b8047349","Type":"ContainerDied","Data":"7d8bc4161e391e1ccf60f3b079c68daaae5941b068616468a5c43a7a28e2f1f2"} Jul 09 23:07:17.971191 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:17.971175 2568 scope.go:117] "RemoveContainer" containerID="7d8bc4161e391e1ccf60f3b079c68daaae5941b068616468a5c43a7a28e2f1f2" Jul 09 23:07:17.972288 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:17.972261 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:07:17.972352 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:17.972330 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:07:17.972390 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:17.972362 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:07:18.981750 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:18.981711 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" event={"ID":"e29eeb17-f328-4403-a427-c0c7b8047349","Type":"ContainerStarted","Data":"3fec625c856da3a195cffc29b770fe7410361edac7d719aae03409f9df31f9a3"} Jul 09 23:07:19.919413 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:19.919373 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:07:19.919665 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:19.919641 2568 patch_prober.go:28] interesting pod/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 09 23:07:19.919716 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:19.919693 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" podUID="e29eeb17-f328-4403-a427-c0c7b8047349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 09 23:07:20.919013 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:20.918976 2568 patch_prober.go:28] interesting pod/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 09 23:07:20.919422 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:20.919032 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" podUID="e29eeb17-f328-4403-a427-c0c7b8047349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 09 23:07:21.911175 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:21.911139 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:21.918809 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:21.918773 2568 patch_prober.go:28] interesting pod/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 09 23:07:21.918995 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:21.918840 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" podUID="e29eeb17-f328-4403-a427-c0c7b8047349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 09 23:07:22.919339 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:22.919291 2568 patch_prober.go:28] interesting pod/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 09 23:07:22.919768 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:22.919368 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" podUID="e29eeb17-f328-4403-a427-c0c7b8047349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 09 23:07:23.509162 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:23.509119 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:23.609196 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:23.609157 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:23.922636 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:23.922605 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:07:23.923092 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:23.922982 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:07:23.923704 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:23.923680 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:07:33.029115 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:33.029080 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:33.129032 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:33.129003 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:50.459442 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:50.459404 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:07:55.902077 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:07:55.902043 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:08:18.456454 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:18.456421 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:08:19.387127 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:19.387087 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:08:19.567226 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:19.567183 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:08:19.992084 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:19.992052 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:08:20.091069 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:20.091019 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:08:20.229832 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:20.229792 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4"] Jul 09 23:08:20.230041 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:20.230016 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" podUID="2e97cc60-4bba-4804-99a3-39edeb824436" containerName="maas-api" containerID="cri-o://10427c26cb0b23eed2001071a861a146a22a846d0ec278fa8e0baf152c543f87" gracePeriod=30 Jul 09 23:08:20.256072 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:20.255968 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:08:20.680839 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:20.680809 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" Jul 09 23:08:20.740793 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:20.740755 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e97cc60-4bba-4804-99a3-39edeb824436-openshift-service-ca\") pod \"2e97cc60-4bba-4804-99a3-39edeb824436\" (UID: \"2e97cc60-4bba-4804-99a3-39edeb824436\") " Jul 09 23:08:20.740793 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:20.740801 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p4rr\" (UniqueName: \"kubernetes.io/projected/2e97cc60-4bba-4804-99a3-39edeb824436-kube-api-access-5p4rr\") pod \"2e97cc60-4bba-4804-99a3-39edeb824436\" (UID: \"2e97cc60-4bba-4804-99a3-39edeb824436\") " Jul 09 23:08:20.741071 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:20.740824 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e97cc60-4bba-4804-99a3-39edeb824436-maas-api-tls\") pod \"2e97cc60-4bba-4804-99a3-39edeb824436\" (UID: \"2e97cc60-4bba-4804-99a3-39edeb824436\") " Jul 09 23:08:20.741265 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:20.741240 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e97cc60-4bba-4804-99a3-39edeb824436-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e97cc60-4bba-4804-99a3-39edeb824436" (UID: "2e97cc60-4bba-4804-99a3-39edeb824436"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:08:20.743110 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:20.743076 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e97cc60-4bba-4804-99a3-39edeb824436-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e97cc60-4bba-4804-99a3-39edeb824436" (UID: "2e97cc60-4bba-4804-99a3-39edeb824436"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:08:20.743225 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:20.743111 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e97cc60-4bba-4804-99a3-39edeb824436-kube-api-access-5p4rr" (OuterVolumeSpecName: "kube-api-access-5p4rr") pod "2e97cc60-4bba-4804-99a3-39edeb824436" (UID: "2e97cc60-4bba-4804-99a3-39edeb824436"). InnerVolumeSpecName "kube-api-access-5p4rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:08:20.842119 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:20.842083 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e97cc60-4bba-4804-99a3-39edeb824436-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:08:20.842119 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:20.842116 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p4rr\" (UniqueName: \"kubernetes.io/projected/2e97cc60-4bba-4804-99a3-39edeb824436-kube-api-access-5p4rr\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:08:20.842328 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:20.842129 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e97cc60-4bba-4804-99a3-39edeb824436-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:08:21.335536 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:21.335476 2568 generic.go:356] "Generic (PLEG): container finished" podID="2e97cc60-4bba-4804-99a3-39edeb824436" containerID="10427c26cb0b23eed2001071a861a146a22a846d0ec278fa8e0baf152c543f87" exitCode=0 Jul 09 23:08:21.335725 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:21.335542 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" Jul 09 23:08:21.335725 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:21.335564 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" event={"ID":"2e97cc60-4bba-4804-99a3-39edeb824436","Type":"ContainerDied","Data":"10427c26cb0b23eed2001071a861a146a22a846d0ec278fa8e0baf152c543f87"} Jul 09 23:08:21.335725 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:21.335620 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4" event={"ID":"2e97cc60-4bba-4804-99a3-39edeb824436","Type":"ContainerDied","Data":"3072ef61727f0717ceba638c1089d46a3d3cdeeae42303acda995f0310ee1c9b"} Jul 09 23:08:21.335725 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:21.335641 2568 scope.go:117] "RemoveContainer" containerID="10427c26cb0b23eed2001071a861a146a22a846d0ec278fa8e0baf152c543f87" Jul 09 23:08:21.345741 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:21.345713 2568 scope.go:117] "RemoveContainer" containerID="10427c26cb0b23eed2001071a861a146a22a846d0ec278fa8e0baf152c543f87" Jul 09 23:08:21.346072 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:08:21.346043 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10427c26cb0b23eed2001071a861a146a22a846d0ec278fa8e0baf152c543f87\": container with ID starting with 10427c26cb0b23eed2001071a861a146a22a846d0ec278fa8e0baf152c543f87 not found: ID does not exist" containerID="10427c26cb0b23eed2001071a861a146a22a846d0ec278fa8e0baf152c543f87" Jul 09 23:08:21.346122 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:21.346085 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10427c26cb0b23eed2001071a861a146a22a846d0ec278fa8e0baf152c543f87"} err="failed to get container status \"10427c26cb0b23eed2001071a861a146a22a846d0ec278fa8e0baf152c543f87\": rpc error: code = NotFound desc = could not find container \"10427c26cb0b23eed2001071a861a146a22a846d0ec278fa8e0baf152c543f87\": container with ID starting with 10427c26cb0b23eed2001071a861a146a22a846d0ec278fa8e0baf152c543f87 not found: ID does not exist" Jul 09 23:08:21.357868 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:21.357830 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4"] Jul 09 23:08:21.360763 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:21.360737 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ed9500-8c7b7fdb8-jcjw4"] Jul 09 23:08:22.050196 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:22.050161 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e97cc60-4bba-4804-99a3-39edeb824436" path="/var/lib/kubelet/pods/2e97cc60-4bba-4804-99a3-39edeb824436/volumes" Jul 09 23:08:25.068614 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:08:25.068580 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:01.916155 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:01.916113 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:02.171348 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:02.171246 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx"] Jul 09 23:09:02.171585 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:02.171533 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" podUID="e29eeb17-f328-4403-a427-c0c7b8047349" containerName="istio-proxy" containerID="cri-o://3fec625c856da3a195cffc29b770fe7410361edac7d719aae03409f9df31f9a3" gracePeriod=30 Jul 09 23:09:02.184134 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:02.184105 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:02.429377 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:02.429238 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:02.594991 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:02.594957 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:03.019185 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.019149 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:03.125408 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.125362 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:03.255125 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.255090 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl"] Jul 09 23:09:03.255337 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.255310 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" podUID="5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b" containerName="maas-api" containerID="cri-o://a4a4cd817625d6681e8824dd6c9d08d25de7f3947cae59d2c6694b167da9eb2f" gracePeriod=30 Jul 09 23:09:03.283161 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.283133 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:03.504514 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.504467 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" Jul 09 23:09:03.538793 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.538697 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29jxt\" (UniqueName: \"kubernetes.io/projected/5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b-kube-api-access-29jxt\") pod \"5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b\" (UID: \"5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b\") " Jul 09 23:09:03.538793 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.538782 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b-maas-api-tls\") pod \"5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b\" (UID: \"5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b\") " Jul 09 23:09:03.539033 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.538841 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b-openshift-service-ca\") pod \"5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b\" (UID: \"5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b\") " Jul 09 23:09:03.539279 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.539252 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b" (UID: "5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:09:03.541050 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.541020 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b" (UID: "5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:09:03.541163 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.541028 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b-kube-api-access-29jxt" (OuterVolumeSpecName: "kube-api-access-29jxt") pod "5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b" (UID: "5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b"). InnerVolumeSpecName "kube-api-access-29jxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:09:03.568505 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.568464 2568 generic.go:356] "Generic (PLEG): container finished" podID="5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b" containerID="a4a4cd817625d6681e8824dd6c9d08d25de7f3947cae59d2c6694b167da9eb2f" exitCode=0 Jul 09 23:09:03.568712 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.568524 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" event={"ID":"5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b","Type":"ContainerDied","Data":"a4a4cd817625d6681e8824dd6c9d08d25de7f3947cae59d2c6694b167da9eb2f"} Jul 09 23:09:03.568712 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.568547 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" Jul 09 23:09:03.568712 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.568592 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl" event={"ID":"5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b","Type":"ContainerDied","Data":"ed35b18cfe1a87a2815bf7779e76363b75aacf899a7f6bf6792ba2afc80c2545"} Jul 09 23:09:03.568712 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.568614 2568 scope.go:117] "RemoveContainer" containerID="a4a4cd817625d6681e8824dd6c9d08d25de7f3947cae59d2c6694b167da9eb2f" Jul 09 23:09:03.578378 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.578357 2568 scope.go:117] "RemoveContainer" containerID="a4a4cd817625d6681e8824dd6c9d08d25de7f3947cae59d2c6694b167da9eb2f" Jul 09 23:09:03.578729 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:09:03.578706 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4a4cd817625d6681e8824dd6c9d08d25de7f3947cae59d2c6694b167da9eb2f\": container with ID starting with a4a4cd817625d6681e8824dd6c9d08d25de7f3947cae59d2c6694b167da9eb2f not found: ID does not exist" containerID="a4a4cd817625d6681e8824dd6c9d08d25de7f3947cae59d2c6694b167da9eb2f" Jul 09 23:09:03.578779 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.578738 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4a4cd817625d6681e8824dd6c9d08d25de7f3947cae59d2c6694b167da9eb2f"} err="failed to get container status \"a4a4cd817625d6681e8824dd6c9d08d25de7f3947cae59d2c6694b167da9eb2f\": rpc error: code = NotFound desc = could not find container \"a4a4cd817625d6681e8824dd6c9d08d25de7f3947cae59d2c6694b167da9eb2f\": container with ID starting with a4a4cd817625d6681e8824dd6c9d08d25de7f3947cae59d2c6694b167da9eb2f not found: ID does not exist" Jul 09 23:09:03.590136 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.590098 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl"] Jul 09 23:09:03.592411 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.592381 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ceba72-54544fbfb8-fd4fl"] Jul 09 23:09:03.639770 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.639733 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29jxt\" (UniqueName: \"kubernetes.io/projected/5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b-kube-api-access-29jxt\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:03.639770 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.639766 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:03.639770 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:03.639776 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:04.048263 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:04.048225 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b" path="/var/lib/kubelet/pods/5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b/volumes" Jul 09 23:09:07.429105 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.429077 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:09:07.469084 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.468997 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29eeb17-f328-4403-a427-c0c7b8047349-istiod-ca-cert\") pod \"e29eeb17-f328-4403-a427-c0c7b8047349\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " Jul 09 23:09:07.469084 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.469035 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-workload-socket\") pod \"e29eeb17-f328-4403-a427-c0c7b8047349\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " Jul 09 23:09:07.469084 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.469065 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29eeb17-f328-4403-a427-c0c7b8047349-istio-podinfo\") pod \"e29eeb17-f328-4403-a427-c0c7b8047349\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " Jul 09 23:09:07.469390 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.469098 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-istio-envoy\") pod \"e29eeb17-f328-4403-a427-c0c7b8047349\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " Jul 09 23:09:07.469390 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.469223 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-istio-data\") pod \"e29eeb17-f328-4403-a427-c0c7b8047349\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " Jul 09 23:09:07.469390 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.469275 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fkpk\" (UniqueName: \"kubernetes.io/projected/e29eeb17-f328-4403-a427-c0c7b8047349-kube-api-access-4fkpk\") pod \"e29eeb17-f328-4403-a427-c0c7b8047349\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " Jul 09 23:09:07.469629 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.469396 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-workload-certs\") pod \"e29eeb17-f328-4403-a427-c0c7b8047349\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " Jul 09 23:09:07.469629 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.469416 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e29eeb17-f328-4403-a427-c0c7b8047349" (UID: "e29eeb17-f328-4403-a427-c0c7b8047349"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:09:07.469629 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.469424 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29eeb17-f328-4403-a427-c0c7b8047349-istio-token\") pod \"e29eeb17-f328-4403-a427-c0c7b8047349\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " Jul 09 23:09:07.469629 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.469481 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29eeb17-f328-4403-a427-c0c7b8047349-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e29eeb17-f328-4403-a427-c0c7b8047349" (UID: "e29eeb17-f328-4403-a427-c0c7b8047349"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:09:07.469629 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.469539 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-credential-socket\") pod \"e29eeb17-f328-4403-a427-c0c7b8047349\" (UID: \"e29eeb17-f328-4403-a427-c0c7b8047349\") " Jul 09 23:09:07.469879 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.469690 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e29eeb17-f328-4403-a427-c0c7b8047349" (UID: "e29eeb17-f328-4403-a427-c0c7b8047349"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:09:07.469945 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.469905 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e29eeb17-f328-4403-a427-c0c7b8047349" (UID: "e29eeb17-f328-4403-a427-c0c7b8047349"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:09:07.470109 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.470085 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e29eeb17-f328-4403-a427-c0c7b8047349-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:07.470212 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.470114 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:07.470212 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.470127 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:07.470438 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.470410 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-istio-data" (OuterVolumeSpecName: "istio-data") pod "e29eeb17-f328-4403-a427-c0c7b8047349" (UID: "e29eeb17-f328-4403-a427-c0c7b8047349"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:09:07.471895 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.471862 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29eeb17-f328-4403-a427-c0c7b8047349-kube-api-access-4fkpk" (OuterVolumeSpecName: "kube-api-access-4fkpk") pod "e29eeb17-f328-4403-a427-c0c7b8047349" (UID: "e29eeb17-f328-4403-a427-c0c7b8047349"). InnerVolumeSpecName "kube-api-access-4fkpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:09:07.472016 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.471974 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29eeb17-f328-4403-a427-c0c7b8047349-istio-token" (OuterVolumeSpecName: "istio-token") pod "e29eeb17-f328-4403-a427-c0c7b8047349" (UID: "e29eeb17-f328-4403-a427-c0c7b8047349"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:09:07.472085 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.472040 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e29eeb17-f328-4403-a427-c0c7b8047349" (UID: "e29eeb17-f328-4403-a427-c0c7b8047349"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:09:07.472236 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.472218 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e29eeb17-f328-4403-a427-c0c7b8047349-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e29eeb17-f328-4403-a427-c0c7b8047349" (UID: "e29eeb17-f328-4403-a427-c0c7b8047349"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 23:09:07.571034 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.570996 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e29eeb17-f328-4403-a427-c0c7b8047349-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:07.571034 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.571025 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:07.571034 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.571036 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e29eeb17-f328-4403-a427-c0c7b8047349-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:07.571034 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.571045 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:07.571318 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.571053 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e29eeb17-f328-4403-a427-c0c7b8047349-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:07.571318 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.571063 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fkpk\" (UniqueName: \"kubernetes.io/projected/e29eeb17-f328-4403-a427-c0c7b8047349-kube-api-access-4fkpk\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:07.591794 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.591762 2568 generic.go:356] "Generic (PLEG): container finished" podID="e29eeb17-f328-4403-a427-c0c7b8047349" containerID="3fec625c856da3a195cffc29b770fe7410361edac7d719aae03409f9df31f9a3" exitCode=0 Jul 09 23:09:07.591942 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.591814 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" event={"ID":"e29eeb17-f328-4403-a427-c0c7b8047349","Type":"ContainerDied","Data":"3fec625c856da3a195cffc29b770fe7410361edac7d719aae03409f9df31f9a3"} Jul 09 23:09:07.591942 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.591827 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" Jul 09 23:09:07.591942 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.591858 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx" event={"ID":"e29eeb17-f328-4403-a427-c0c7b8047349","Type":"ContainerDied","Data":"e8a2b735e77f42be56bb2e17d8c747e926587afbb62c536a42211e6a8936efb7"} Jul 09 23:09:07.591942 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.591880 2568 scope.go:117] "RemoveContainer" containerID="3fec625c856da3a195cffc29b770fe7410361edac7d719aae03409f9df31f9a3" Jul 09 23:09:07.601661 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.601642 2568 scope.go:117] "RemoveContainer" containerID="7d8bc4161e391e1ccf60f3b079c68daaae5941b068616468a5c43a7a28e2f1f2" Jul 09 23:09:07.610157 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.610130 2568 scope.go:117] "RemoveContainer" containerID="3fec625c856da3a195cffc29b770fe7410361edac7d719aae03409f9df31f9a3" Jul 09 23:09:07.610432 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:09:07.610413 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fec625c856da3a195cffc29b770fe7410361edac7d719aae03409f9df31f9a3\": container with ID starting with 3fec625c856da3a195cffc29b770fe7410361edac7d719aae03409f9df31f9a3 not found: ID does not exist" containerID="3fec625c856da3a195cffc29b770fe7410361edac7d719aae03409f9df31f9a3" Jul 09 23:09:07.610500 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.610442 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fec625c856da3a195cffc29b770fe7410361edac7d719aae03409f9df31f9a3"} err="failed to get container status \"3fec625c856da3a195cffc29b770fe7410361edac7d719aae03409f9df31f9a3\": rpc error: code = NotFound desc = could not find container \"3fec625c856da3a195cffc29b770fe7410361edac7d719aae03409f9df31f9a3\": container with ID starting with 3fec625c856da3a195cffc29b770fe7410361edac7d719aae03409f9df31f9a3 not found: ID does not exist" Jul 09 23:09:07.610500 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.610459 2568 scope.go:117] "RemoveContainer" containerID="7d8bc4161e391e1ccf60f3b079c68daaae5941b068616468a5c43a7a28e2f1f2" Jul 09 23:09:07.610737 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:09:07.610720 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d8bc4161e391e1ccf60f3b079c68daaae5941b068616468a5c43a7a28e2f1f2\": container with ID starting with 7d8bc4161e391e1ccf60f3b079c68daaae5941b068616468a5c43a7a28e2f1f2 not found: ID does not exist" containerID="7d8bc4161e391e1ccf60f3b079c68daaae5941b068616468a5c43a7a28e2f1f2" Jul 09 23:09:07.610847 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.610740 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d8bc4161e391e1ccf60f3b079c68daaae5941b068616468a5c43a7a28e2f1f2"} err="failed to get container status \"7d8bc4161e391e1ccf60f3b079c68daaae5941b068616468a5c43a7a28e2f1f2\": rpc error: code = NotFound desc = could not find container \"7d8bc4161e391e1ccf60f3b079c68daaae5941b068616468a5c43a7a28e2f1f2\": container with ID starting with 7d8bc4161e391e1ccf60f3b079c68daaae5941b068616468a5c43a7a28e2f1f2 not found: ID does not exist" Jul 09 23:09:07.614287 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.614257 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx"] Jul 09 23:09:07.618238 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:07.618211 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ed9500-openshift-default-97b64f99f-w9knx"] Jul 09 23:09:08.054028 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:08.053990 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e29eeb17-f328-4403-a427-c0c7b8047349" path="/var/lib/kubelet/pods/e29eeb17-f328-4403-a427-c0c7b8047349/volumes" Jul 09 23:09:08.607530 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:08.607471 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:08.672115 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:08.672081 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:08.915202 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:08.915114 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:09.141036 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.141002 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:09.486135 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.486101 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:09.767107 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.767022 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp"] Jul 09 23:09:09.767497 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.767386 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b" containerName="maas-api" Jul 09 23:09:09.767497 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.767401 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b" containerName="maas-api" Jul 09 23:09:09.767497 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.767414 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e29eeb17-f328-4403-a427-c0c7b8047349" containerName="istio-proxy" Jul 09 23:09:09.767497 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.767420 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e29eeb17-f328-4403-a427-c0c7b8047349" containerName="istio-proxy" Jul 09 23:09:09.767497 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.767434 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e97cc60-4bba-4804-99a3-39edeb824436" containerName="maas-api" Jul 09 23:09:09.767497 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.767440 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e97cc60-4bba-4804-99a3-39edeb824436" containerName="maas-api" Jul 09 23:09:09.767735 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.767523 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e97cc60-4bba-4804-99a3-39edeb824436" containerName="maas-api" Jul 09 23:09:09.767735 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.767533 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e29eeb17-f328-4403-a427-c0c7b8047349" containerName="istio-proxy" Jul 09 23:09:09.767735 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.767542 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d5ee1ba-3dd8-4ac3-a2ae-f1985f54985b" containerName="maas-api" Jul 09 23:09:09.771987 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.771966 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.774886 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.774861 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-728627-openshift-default-dockercfg-bl487\"" Jul 09 23:09:09.781248 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.781220 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp"] Jul 09 23:09:09.786980 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.786951 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:09.793268 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.793242 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-envoy\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.793380 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.793279 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-token\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.793380 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.793308 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-credential-socket\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.793480 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.793382 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-data\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.793480 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.793467 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-workload-socket\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.793575 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.793521 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-workload-certs\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.793575 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.793547 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp8jr\" (UniqueName: \"kubernetes.io/projected/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-kube-api-access-hp8jr\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.793652 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.793587 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istiod-ca-cert\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.793652 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.793629 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-podinfo\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.894840 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.894799 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-data\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.895026 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.894854 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-workload-socket\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.895026 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.894872 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-workload-certs\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.895026 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.894892 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp8jr\" (UniqueName: \"kubernetes.io/projected/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-kube-api-access-hp8jr\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.895026 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.894917 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istiod-ca-cert\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.895026 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.894944 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-podinfo\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.895026 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.894961 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-envoy\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.895026 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.894978 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-token\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.895026 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.895010 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-credential-socket\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.895411 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.895267 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-data\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.895411 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.895322 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-workload-socket\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.895411 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.895341 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-workload-certs\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.895611 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.895423 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-credential-socket\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.895668 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.895629 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istiod-ca-cert\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.897413 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.897392 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-podinfo\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.897505 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.897415 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-envoy\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.903330 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.903304 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp8jr\" (UniqueName: \"kubernetes.io/projected/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-kube-api-access-hp8jr\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:09.903434 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:09.903345 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-token\") pod \"e2e-api-a-728627-openshift-default-f4d9d675f-24fxp\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:10.023320 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:10.023232 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:10.083821 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:10.083784 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:10.223768 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:10.223735 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp"] Jul 09 23:09:10.226442 ip-10-0-135-80 kubenswrapper[2568]: W0709 23:09:10.226403 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ca8b55_a523_4c15_ac8f_70a5939d41bb.slice/crio-2945a216f3cc4fc68df680f8bdd9cfb9a1a4e1aa4ee4d9b1eb92b5ae661a8f09 WatchSource:0}: Error finding container 2945a216f3cc4fc68df680f8bdd9cfb9a1a4e1aa4ee4d9b1eb92b5ae661a8f09: Status 404 returned error can't find the container with id 2945a216f3cc4fc68df680f8bdd9cfb9a1a4e1aa4ee4d9b1eb92b5ae661a8f09 Jul 09 23:09:10.228615 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:10.228579 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:09:10.228728 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:10.228662 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:09:10.228728 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:10.228706 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:09:10.610615 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:10.610583 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" event={"ID":"e6ca8b55-a523-4c15-ac8f-70a5939d41bb","Type":"ContainerStarted","Data":"217a9cf9cdccf2b26b278e28b4d0a2f2bf09a2f05f2db82f20ff138f73d923c6"} Jul 09 23:09:10.610797 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:10.610630 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" event={"ID":"e6ca8b55-a523-4c15-ac8f-70a5939d41bb","Type":"ContainerStarted","Data":"2945a216f3cc4fc68df680f8bdd9cfb9a1a4e1aa4ee4d9b1eb92b5ae661a8f09"} Jul 09 23:09:10.636395 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:10.636337 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" podStartSLOduration=1.636314378 podStartE2EDuration="1.636314378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:09:10.635598351 +0000 UTC m=+2175.176677157" watchObservedRunningTime="2026-07-09 23:09:10.636314378 +0000 UTC m=+2175.177393173" Jul 09 23:09:11.084464 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:11.084430 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:11.089575 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:11.089552 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:11.615375 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:11.615347 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:11.616235 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:11.616214 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:12.604865 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.604812 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:12.670676 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.670638 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9"] Jul 09 23:09:12.671137 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.671122 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e29eeb17-f328-4403-a427-c0c7b8047349" containerName="istio-proxy" Jul 09 23:09:12.671206 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.671139 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e29eeb17-f328-4403-a427-c0c7b8047349" containerName="istio-proxy" Jul 09 23:09:12.671248 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.671206 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e29eeb17-f328-4403-a427-c0c7b8047349" containerName="istio-proxy" Jul 09 23:09:12.674288 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.674268 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" Jul 09 23:09:12.677095 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.677068 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-728627-serving-cert\"" Jul 09 23:09:12.684728 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.684697 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9"] Jul 09 23:09:12.698047 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.698011 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:12.721099 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.721043 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlg5b\" (UniqueName: \"kubernetes.io/projected/aa642935-9a53-4abd-a54a-6e616259e1b2-kube-api-access-jlg5b\") pod \"maas-api-e2e-api-a-728627-ffd799694-s4kv9\" (UID: \"aa642935-9a53-4abd-a54a-6e616259e1b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" Jul 09 23:09:12.721099 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.721094 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa642935-9a53-4abd-a54a-6e616259e1b2-openshift-service-ca\") pod \"maas-api-e2e-api-a-728627-ffd799694-s4kv9\" (UID: \"aa642935-9a53-4abd-a54a-6e616259e1b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" Jul 09 23:09:12.721359 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.721334 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa642935-9a53-4abd-a54a-6e616259e1b2-maas-api-tls\") pod \"maas-api-e2e-api-a-728627-ffd799694-s4kv9\" (UID: \"aa642935-9a53-4abd-a54a-6e616259e1b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" Jul 09 23:09:12.757924 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.757888 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:12.822540 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.822474 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa642935-9a53-4abd-a54a-6e616259e1b2-openshift-service-ca\") pod \"maas-api-e2e-api-a-728627-ffd799694-s4kv9\" (UID: \"aa642935-9a53-4abd-a54a-6e616259e1b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" Jul 09 23:09:12.822713 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.822599 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa642935-9a53-4abd-a54a-6e616259e1b2-maas-api-tls\") pod \"maas-api-e2e-api-a-728627-ffd799694-s4kv9\" (UID: \"aa642935-9a53-4abd-a54a-6e616259e1b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" Jul 09 23:09:12.822713 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.822648 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlg5b\" (UniqueName: \"kubernetes.io/projected/aa642935-9a53-4abd-a54a-6e616259e1b2-kube-api-access-jlg5b\") pod \"maas-api-e2e-api-a-728627-ffd799694-s4kv9\" (UID: \"aa642935-9a53-4abd-a54a-6e616259e1b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" Jul 09 23:09:12.823131 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.823105 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa642935-9a53-4abd-a54a-6e616259e1b2-openshift-service-ca\") pod \"maas-api-e2e-api-a-728627-ffd799694-s4kv9\" (UID: \"aa642935-9a53-4abd-a54a-6e616259e1b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" Jul 09 23:09:12.825109 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.825084 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa642935-9a53-4abd-a54a-6e616259e1b2-maas-api-tls\") pod \"maas-api-e2e-api-a-728627-ffd799694-s4kv9\" (UID: \"aa642935-9a53-4abd-a54a-6e616259e1b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" Jul 09 23:09:12.830414 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.830391 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlg5b\" (UniqueName: \"kubernetes.io/projected/aa642935-9a53-4abd-a54a-6e616259e1b2-kube-api-access-jlg5b\") pod \"maas-api-e2e-api-a-728627-ffd799694-s4kv9\" (UID: \"aa642935-9a53-4abd-a54a-6e616259e1b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" Jul 09 23:09:12.859722 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.859627 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:12.986380 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:12.986346 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" Jul 09 23:09:13.129304 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:13.129278 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9"] Jul 09 23:09:13.131070 ip-10-0-135-80 kubenswrapper[2568]: W0709 23:09:13.131042 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa642935_9a53_4abd_a54a_6e616259e1b2.slice/crio-40fb9a9682370e48e19a49d95669101ebef04d6edb3169fa418bec2670eb658c WatchSource:0}: Error finding container 40fb9a9682370e48e19a49d95669101ebef04d6edb3169fa418bec2670eb658c: Status 404 returned error can't find the container with id 40fb9a9682370e48e19a49d95669101ebef04d6edb3169fa418bec2670eb658c Jul 09 23:09:13.149068 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:13.149033 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:13.253925 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:13.253894 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:13.627337 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:13.627301 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" event={"ID":"aa642935-9a53-4abd-a54a-6e616259e1b2","Type":"ContainerStarted","Data":"4cd44437be5518df89292da274f8a96dbba120b1362137ef9743da7d7077df7a"} Jul 09 23:09:13.627760 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:13.627358 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" event={"ID":"aa642935-9a53-4abd-a54a-6e616259e1b2","Type":"ContainerStarted","Data":"40fb9a9682370e48e19a49d95669101ebef04d6edb3169fa418bec2670eb658c"} Jul 09 23:09:13.644906 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:13.644856 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" podStartSLOduration=1.254486446 podStartE2EDuration="1.64484146s" totalImagesPullingTime="390.355014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:09:13.643618129 +0000 UTC m=+2178.184696925" watchObservedRunningTime="2026-07-09 23:09:13.64484146 +0000 UTC m=+2178.185920253" Jul 09 23:09:14.632833 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:14.632809 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" Jul 09 23:09:17.115200 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:17.115170 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:17.588102 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:17.588068 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:17.911223 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:17.911138 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:18.163000 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:18.162905 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:20.643281 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:20.643252 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" Jul 09 23:09:20.896187 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:20.896088 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:20.964170 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:20.964129 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w"] Jul 09 23:09:20.967941 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:20.967912 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" Jul 09 23:09:20.970485 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:20.970461 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-669c32-serving-cert\"" Jul 09 23:09:20.982336 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:20.982300 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w"] Jul 09 23:09:20.987889 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:20.987863 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:21.056183 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:21.056138 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:21.097767 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:21.097733 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25cacd18-8950-4448-b39f-7d11275241b4-maas-api-tls\") pod \"maas-api-e2e-api-b-669c32-86d9488d6d-q764w\" (UID: \"25cacd18-8950-4448-b39f-7d11275241b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" Jul 09 23:09:21.097938 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:21.097778 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cacd18-8950-4448-b39f-7d11275241b4-openshift-service-ca\") pod \"maas-api-e2e-api-b-669c32-86d9488d6d-q764w\" (UID: \"25cacd18-8950-4448-b39f-7d11275241b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" Jul 09 23:09:21.097938 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:21.097906 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fmzz\" (UniqueName: \"kubernetes.io/projected/25cacd18-8950-4448-b39f-7d11275241b4-kube-api-access-7fmzz\") pod \"maas-api-e2e-api-b-669c32-86d9488d6d-q764w\" (UID: \"25cacd18-8950-4448-b39f-7d11275241b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" Jul 09 23:09:21.181560 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:21.181444 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:21.198594 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:21.198560 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cacd18-8950-4448-b39f-7d11275241b4-openshift-service-ca\") pod \"maas-api-e2e-api-b-669c32-86d9488d6d-q764w\" (UID: \"25cacd18-8950-4448-b39f-7d11275241b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" Jul 09 23:09:21.198722 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:21.198648 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fmzz\" (UniqueName: \"kubernetes.io/projected/25cacd18-8950-4448-b39f-7d11275241b4-kube-api-access-7fmzz\") pod \"maas-api-e2e-api-b-669c32-86d9488d6d-q764w\" (UID: \"25cacd18-8950-4448-b39f-7d11275241b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" Jul 09 23:09:21.198722 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:21.198693 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25cacd18-8950-4448-b39f-7d11275241b4-maas-api-tls\") pod \"maas-api-e2e-api-b-669c32-86d9488d6d-q764w\" (UID: \"25cacd18-8950-4448-b39f-7d11275241b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" Jul 09 23:09:21.199288 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:21.199266 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cacd18-8950-4448-b39f-7d11275241b4-openshift-service-ca\") pod \"maas-api-e2e-api-b-669c32-86d9488d6d-q764w\" (UID: \"25cacd18-8950-4448-b39f-7d11275241b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" Jul 09 23:09:21.201236 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:21.201212 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25cacd18-8950-4448-b39f-7d11275241b4-maas-api-tls\") pod \"maas-api-e2e-api-b-669c32-86d9488d6d-q764w\" (UID: \"25cacd18-8950-4448-b39f-7d11275241b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" Jul 09 23:09:21.207546 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:21.207521 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fmzz\" (UniqueName: \"kubernetes.io/projected/25cacd18-8950-4448-b39f-7d11275241b4-kube-api-access-7fmzz\") pod \"maas-api-e2e-api-b-669c32-86d9488d6d-q764w\" (UID: \"25cacd18-8950-4448-b39f-7d11275241b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" Jul 09 23:09:21.279917 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:21.279884 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" Jul 09 23:09:21.423668 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:21.423629 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w"] Jul 09 23:09:21.427326 ip-10-0-135-80 kubenswrapper[2568]: W0709 23:09:21.427292 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cacd18_8950_4448_b39f_7d11275241b4.slice/crio-a6f6a09d04e483a9525903df7d1afe5ba5d5824033048187d3be43d4699d1366 WatchSource:0}: Error finding container a6f6a09d04e483a9525903df7d1afe5ba5d5824033048187d3be43d4699d1366: Status 404 returned error can't find the container with id a6f6a09d04e483a9525903df7d1afe5ba5d5824033048187d3be43d4699d1366 Jul 09 23:09:21.543648 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:21.543563 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:21.644967 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:21.644938 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:21.672772 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:21.672736 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" event={"ID":"25cacd18-8950-4448-b39f-7d11275241b4","Type":"ContainerStarted","Data":"a6f6a09d04e483a9525903df7d1afe5ba5d5824033048187d3be43d4699d1366"} Jul 09 23:09:22.684058 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:22.684011 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" event={"ID":"25cacd18-8950-4448-b39f-7d11275241b4","Type":"ContainerStarted","Data":"997f451abb9c5e199a20d1f965583e2729af78f2daafb84641198a3d34b3fe39"} Jul 09 23:09:22.684889 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:22.684862 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" Jul 09 23:09:22.703810 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:22.703753 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" podStartSLOduration=2.330338812 podStartE2EDuration="2.7037347s" totalImagesPullingTime="373.395888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:09:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:09:22.701561788 +0000 UTC m=+2187.242640606" watchObservedRunningTime="2026-07-09 23:09:22.7037347 +0000 UTC m=+2187.244813494" Jul 09 23:09:25.412539 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:25.412506 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:27.657680 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:27.657615 2568 patch_prober.go:28] interesting pod/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 23:09:27.658163 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:27.657677 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" podUID="e6ca8b55-a523-4c15-ac8f-70a5939d41bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 23:09:29.701135 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:29.701100 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" Jul 09 23:09:29.724925 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:29.724890 2568 generic.go:356] "Generic (PLEG): container finished" podID="e6ca8b55-a523-4c15-ac8f-70a5939d41bb" containerID="217a9cf9cdccf2b26b278e28b4d0a2f2bf09a2f05f2db82f20ff138f73d923c6" exitCode=0 Jul 09 23:09:29.725140 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:29.724970 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" event={"ID":"e6ca8b55-a523-4c15-ac8f-70a5939d41bb","Type":"ContainerDied","Data":"217a9cf9cdccf2b26b278e28b4d0a2f2bf09a2f05f2db82f20ff138f73d923c6"} Jul 09 23:09:29.725410 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:29.725392 2568 scope.go:117] "RemoveContainer" containerID="217a9cf9cdccf2b26b278e28b4d0a2f2bf09a2f05f2db82f20ff138f73d923c6" Jul 09 23:09:29.726825 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:29.726653 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:09:29.726825 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:29.726734 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:09:29.726825 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:29.726776 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:09:30.732407 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:30.732370 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" event={"ID":"e6ca8b55-a523-4c15-ac8f-70a5939d41bb","Type":"ContainerStarted","Data":"b688c3d149c912bd4bb9fcdc7fa40034120f1ce5871405625553388dc79ff017"} Jul 09 23:09:31.084430 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:31.084391 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:31.084791 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:31.084764 2568 patch_prober.go:28] interesting pod/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 09 23:09:31.084971 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:31.084943 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" podUID="e6ca8b55-a523-4c15-ac8f-70a5939d41bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 09 23:09:32.084393 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:32.084340 2568 patch_prober.go:28] interesting pod/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 09 23:09:32.084953 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:32.084424 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" podUID="e6ca8b55-a523-4c15-ac8f-70a5939d41bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 09 23:09:33.085087 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:33.085043 2568 patch_prober.go:28] interesting pod/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 09 23:09:33.085658 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:33.085581 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" podUID="e6ca8b55-a523-4c15-ac8f-70a5939d41bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 09 23:09:33.592351 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:33.592311 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:33.992725 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:33.992641 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:34.084327 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.084284 2568 patch_prober.go:28] interesting pod/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 09 23:09:34.084612 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.084583 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" podUID="e6ca8b55-a523-4c15-ac8f-70a5939d41bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 09 23:09:34.095954 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.095922 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:34.194243 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.194190 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:34.303086 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.302983 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9"] Jul 09 23:09:34.303302 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.303220 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" podUID="aa642935-9a53-4abd-a54a-6e616259e1b2" containerName="maas-api" containerID="cri-o://4cd44437be5518df89292da274f8a96dbba120b1362137ef9743da7d7077df7a" gracePeriod=30 Jul 09 23:09:34.303415 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.303230 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:34.598075 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.598045 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" Jul 09 23:09:34.726857 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.726820 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlg5b\" (UniqueName: \"kubernetes.io/projected/aa642935-9a53-4abd-a54a-6e616259e1b2-kube-api-access-jlg5b\") pod \"aa642935-9a53-4abd-a54a-6e616259e1b2\" (UID: \"aa642935-9a53-4abd-a54a-6e616259e1b2\") " Jul 09 23:09:34.727014 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.726890 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa642935-9a53-4abd-a54a-6e616259e1b2-maas-api-tls\") pod \"aa642935-9a53-4abd-a54a-6e616259e1b2\" (UID: \"aa642935-9a53-4abd-a54a-6e616259e1b2\") " Jul 09 23:09:34.727014 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.726938 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa642935-9a53-4abd-a54a-6e616259e1b2-openshift-service-ca\") pod \"aa642935-9a53-4abd-a54a-6e616259e1b2\" (UID: \"aa642935-9a53-4abd-a54a-6e616259e1b2\") " Jul 09 23:09:34.727312 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.727282 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa642935-9a53-4abd-a54a-6e616259e1b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aa642935-9a53-4abd-a54a-6e616259e1b2" (UID: "aa642935-9a53-4abd-a54a-6e616259e1b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:09:34.729162 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.729132 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa642935-9a53-4abd-a54a-6e616259e1b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aa642935-9a53-4abd-a54a-6e616259e1b2" (UID: "aa642935-9a53-4abd-a54a-6e616259e1b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:09:34.729162 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.729155 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa642935-9a53-4abd-a54a-6e616259e1b2-kube-api-access-jlg5b" (OuterVolumeSpecName: "kube-api-access-jlg5b") pod "aa642935-9a53-4abd-a54a-6e616259e1b2" (UID: "aa642935-9a53-4abd-a54a-6e616259e1b2"). InnerVolumeSpecName "kube-api-access-jlg5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:09:34.756073 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.756036 2568 generic.go:356] "Generic (PLEG): container finished" podID="aa642935-9a53-4abd-a54a-6e616259e1b2" containerID="4cd44437be5518df89292da274f8a96dbba120b1362137ef9743da7d7077df7a" exitCode=0 Jul 09 23:09:34.756266 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.756084 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" Jul 09 23:09:34.756266 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.756136 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" event={"ID":"aa642935-9a53-4abd-a54a-6e616259e1b2","Type":"ContainerDied","Data":"4cd44437be5518df89292da274f8a96dbba120b1362137ef9743da7d7077df7a"} Jul 09 23:09:34.756266 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.756189 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9" event={"ID":"aa642935-9a53-4abd-a54a-6e616259e1b2","Type":"ContainerDied","Data":"40fb9a9682370e48e19a49d95669101ebef04d6edb3169fa418bec2670eb658c"} Jul 09 23:09:34.756266 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.756212 2568 scope.go:117] "RemoveContainer" containerID="4cd44437be5518df89292da274f8a96dbba120b1362137ef9743da7d7077df7a" Jul 09 23:09:34.780391 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.780362 2568 scope.go:117] "RemoveContainer" containerID="4cd44437be5518df89292da274f8a96dbba120b1362137ef9743da7d7077df7a" Jul 09 23:09:34.780736 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:09:34.780715 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cd44437be5518df89292da274f8a96dbba120b1362137ef9743da7d7077df7a\": container with ID starting with 4cd44437be5518df89292da274f8a96dbba120b1362137ef9743da7d7077df7a not found: ID does not exist" containerID="4cd44437be5518df89292da274f8a96dbba120b1362137ef9743da7d7077df7a" Jul 09 23:09:34.780847 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.780746 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cd44437be5518df89292da274f8a96dbba120b1362137ef9743da7d7077df7a"} err="failed to get container status \"4cd44437be5518df89292da274f8a96dbba120b1362137ef9743da7d7077df7a\": rpc error: code = NotFound desc = could not find container \"4cd44437be5518df89292da274f8a96dbba120b1362137ef9743da7d7077df7a\": container with ID starting with 4cd44437be5518df89292da274f8a96dbba120b1362137ef9743da7d7077df7a not found: ID does not exist" Jul 09 23:09:34.793030 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.792992 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9"] Jul 09 23:09:34.796861 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.796830 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-728627-ffd799694-s4kv9"] Jul 09 23:09:34.827999 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.827908 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlg5b\" (UniqueName: \"kubernetes.io/projected/aa642935-9a53-4abd-a54a-6e616259e1b2-kube-api-access-jlg5b\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:34.827999 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.827939 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa642935-9a53-4abd-a54a-6e616259e1b2-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:34.827999 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:34.827952 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa642935-9a53-4abd-a54a-6e616259e1b2-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:35.083920 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:35.083800 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:35.087309 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:35.087282 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:35.087572 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:35.087548 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:35.088164 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:35.088149 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:35.184452 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:35.184415 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:36.054042 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:36.054009 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa642935-9a53-4abd-a54a-6e616259e1b2" path="/var/lib/kubelet/pods/aa642935-9a53-4abd-a54a-6e616259e1b2/volumes" Jul 09 23:09:39.975515 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:39.975469 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:40.075575 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:40.075539 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:40.215892 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:40.215859 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp"] Jul 09 23:09:40.216085 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:40.216061 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" podUID="e6ca8b55-a523-4c15-ac8f-70a5939d41bb" containerName="istio-proxy" containerID="cri-o://b688c3d149c912bd4bb9fcdc7fa40034120f1ce5871405625553388dc79ff017" gracePeriod=30 Jul 09 23:09:40.240410 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:40.240311 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:40.509420 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:40.509315 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:40.667927 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:40.667897 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:41.091832 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.091796 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:41.249997 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.249954 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:41.343094 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.343009 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:41.476825 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.476787 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w"] Jul 09 23:09:41.477102 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.477065 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" podUID="25cacd18-8950-4448-b39f-7d11275241b4" containerName="maas-api" containerID="cri-o://997f451abb9c5e199a20d1f965583e2729af78f2daafb84641198a3d34b3fe39" gracePeriod=30 Jul 09 23:09:41.504061 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.504018 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:41.725066 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.725040 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" Jul 09 23:09:41.795370 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.795341 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fmzz\" (UniqueName: \"kubernetes.io/projected/25cacd18-8950-4448-b39f-7d11275241b4-kube-api-access-7fmzz\") pod \"25cacd18-8950-4448-b39f-7d11275241b4\" (UID: \"25cacd18-8950-4448-b39f-7d11275241b4\") " Jul 09 23:09:41.795597 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.795406 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25cacd18-8950-4448-b39f-7d11275241b4-maas-api-tls\") pod \"25cacd18-8950-4448-b39f-7d11275241b4\" (UID: \"25cacd18-8950-4448-b39f-7d11275241b4\") " Jul 09 23:09:41.795597 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.795467 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cacd18-8950-4448-b39f-7d11275241b4-openshift-service-ca\") pod \"25cacd18-8950-4448-b39f-7d11275241b4\" (UID: \"25cacd18-8950-4448-b39f-7d11275241b4\") " Jul 09 23:09:41.795933 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.795900 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25cacd18-8950-4448-b39f-7d11275241b4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "25cacd18-8950-4448-b39f-7d11275241b4" (UID: "25cacd18-8950-4448-b39f-7d11275241b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:09:41.796054 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.795983 2568 generic.go:356] "Generic (PLEG): container finished" podID="25cacd18-8950-4448-b39f-7d11275241b4" containerID="997f451abb9c5e199a20d1f965583e2729af78f2daafb84641198a3d34b3fe39" exitCode=0 Jul 09 23:09:41.796054 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.796031 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" Jul 09 23:09:41.796197 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.796061 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" event={"ID":"25cacd18-8950-4448-b39f-7d11275241b4","Type":"ContainerDied","Data":"997f451abb9c5e199a20d1f965583e2729af78f2daafb84641198a3d34b3fe39"} Jul 09 23:09:41.796197 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.796119 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w" event={"ID":"25cacd18-8950-4448-b39f-7d11275241b4","Type":"ContainerDied","Data":"a6f6a09d04e483a9525903df7d1afe5ba5d5824033048187d3be43d4699d1366"} Jul 09 23:09:41.796197 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.796149 2568 scope.go:117] "RemoveContainer" containerID="997f451abb9c5e199a20d1f965583e2729af78f2daafb84641198a3d34b3fe39" Jul 09 23:09:41.797879 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.797844 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cacd18-8950-4448-b39f-7d11275241b4-kube-api-access-7fmzz" (OuterVolumeSpecName: "kube-api-access-7fmzz") pod "25cacd18-8950-4448-b39f-7d11275241b4" (UID: "25cacd18-8950-4448-b39f-7d11275241b4"). InnerVolumeSpecName "kube-api-access-7fmzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:09:41.797985 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.797946 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25cacd18-8950-4448-b39f-7d11275241b4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "25cacd18-8950-4448-b39f-7d11275241b4" (UID: "25cacd18-8950-4448-b39f-7d11275241b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:09:41.817219 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.817185 2568 scope.go:117] "RemoveContainer" containerID="997f451abb9c5e199a20d1f965583e2729af78f2daafb84641198a3d34b3fe39" Jul 09 23:09:41.817888 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:09:41.817860 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997f451abb9c5e199a20d1f965583e2729af78f2daafb84641198a3d34b3fe39\": container with ID starting with 997f451abb9c5e199a20d1f965583e2729af78f2daafb84641198a3d34b3fe39 not found: ID does not exist" containerID="997f451abb9c5e199a20d1f965583e2729af78f2daafb84641198a3d34b3fe39" Jul 09 23:09:41.818001 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.817895 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997f451abb9c5e199a20d1f965583e2729af78f2daafb84641198a3d34b3fe39"} err="failed to get container status \"997f451abb9c5e199a20d1f965583e2729af78f2daafb84641198a3d34b3fe39\": rpc error: code = NotFound desc = could not find container \"997f451abb9c5e199a20d1f965583e2729af78f2daafb84641198a3d34b3fe39\": container with ID starting with 997f451abb9c5e199a20d1f965583e2729af78f2daafb84641198a3d34b3fe39 not found: ID does not exist" Jul 09 23:09:41.896948 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.896846 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cacd18-8950-4448-b39f-7d11275241b4-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:41.896948 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.896881 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fmzz\" (UniqueName: \"kubernetes.io/projected/25cacd18-8950-4448-b39f-7d11275241b4-kube-api-access-7fmzz\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:41.896948 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:41.896890 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25cacd18-8950-4448-b39f-7d11275241b4-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:42.115610 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:42.115551 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w"] Jul 09 23:09:42.119025 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:42.119000 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-669c32-86d9488d6d-q764w"] Jul 09 23:09:44.048252 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:44.048212 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25cacd18-8950-4448-b39f-7d11275241b4" path="/var/lib/kubelet/pods/25cacd18-8950-4448-b39f-7d11275241b4/volumes" Jul 09 23:09:45.473106 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.473076 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:45.633097 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.633048 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-workload-socket\") pod \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " Jul 09 23:09:45.633097 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.633102 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp8jr\" (UniqueName: \"kubernetes.io/projected/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-kube-api-access-hp8jr\") pod \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " Jul 09 23:09:45.633378 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.633132 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-token\") pod \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " Jul 09 23:09:45.633378 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.633150 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-podinfo\") pod \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " Jul 09 23:09:45.633378 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.633169 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-workload-certs\") pod \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " Jul 09 23:09:45.633378 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.633202 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-credential-socket\") pod \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " Jul 09 23:09:45.633378 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.633251 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istiod-ca-cert\") pod \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " Jul 09 23:09:45.633378 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.633299 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-envoy\") pod \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " Jul 09 23:09:45.633378 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.633338 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-data\") pod \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\" (UID: \"e6ca8b55-a523-4c15-ac8f-70a5939d41bb\") " Jul 09 23:09:45.633798 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.633388 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e6ca8b55-a523-4c15-ac8f-70a5939d41bb" (UID: "e6ca8b55-a523-4c15-ac8f-70a5939d41bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:09:45.633798 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.633588 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e6ca8b55-a523-4c15-ac8f-70a5939d41bb" (UID: "e6ca8b55-a523-4c15-ac8f-70a5939d41bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:09:45.633798 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.633655 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:45.633798 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.633671 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:45.634011 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.633805 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e6ca8b55-a523-4c15-ac8f-70a5939d41bb" (UID: "e6ca8b55-a523-4c15-ac8f-70a5939d41bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:09:45.634011 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.633847 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e6ca8b55-a523-4c15-ac8f-70a5939d41bb" (UID: "e6ca8b55-a523-4c15-ac8f-70a5939d41bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:09:45.634318 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.634297 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "e6ca8b55-a523-4c15-ac8f-70a5939d41bb" (UID: "e6ca8b55-a523-4c15-ac8f-70a5939d41bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:09:45.635618 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.635588 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-kube-api-access-hp8jr" (OuterVolumeSpecName: "kube-api-access-hp8jr") pod "e6ca8b55-a523-4c15-ac8f-70a5939d41bb" (UID: "e6ca8b55-a523-4c15-ac8f-70a5939d41bb"). InnerVolumeSpecName "kube-api-access-hp8jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:09:45.635618 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.635598 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "e6ca8b55-a523-4c15-ac8f-70a5939d41bb" (UID: "e6ca8b55-a523-4c15-ac8f-70a5939d41bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:09:45.635873 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.635857 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e6ca8b55-a523-4c15-ac8f-70a5939d41bb" (UID: "e6ca8b55-a523-4c15-ac8f-70a5939d41bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 23:09:45.635936 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.635897 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e6ca8b55-a523-4c15-ac8f-70a5939d41bb" (UID: "e6ca8b55-a523-4c15-ac8f-70a5939d41bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:09:45.734611 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.734570 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp8jr\" (UniqueName: \"kubernetes.io/projected/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-kube-api-access-hp8jr\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:45.734611 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.734604 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:45.734611 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.734613 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:45.734611 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.734621 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:45.735258 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.734630 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:45.735258 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.734638 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:45.735258 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.734646 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6ca8b55-a523-4c15-ac8f-70a5939d41bb-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:09:45.820795 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.820762 2568 generic.go:356] "Generic (PLEG): container finished" podID="e6ca8b55-a523-4c15-ac8f-70a5939d41bb" containerID="b688c3d149c912bd4bb9fcdc7fa40034120f1ce5871405625553388dc79ff017" exitCode=0 Jul 09 23:09:45.820968 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.820815 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" Jul 09 23:09:45.820968 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.820853 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" event={"ID":"e6ca8b55-a523-4c15-ac8f-70a5939d41bb","Type":"ContainerDied","Data":"b688c3d149c912bd4bb9fcdc7fa40034120f1ce5871405625553388dc79ff017"} Jul 09 23:09:45.820968 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.820906 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp" event={"ID":"e6ca8b55-a523-4c15-ac8f-70a5939d41bb","Type":"ContainerDied","Data":"2945a216f3cc4fc68df680f8bdd9cfb9a1a4e1aa4ee4d9b1eb92b5ae661a8f09"} Jul 09 23:09:45.820968 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.820928 2568 scope.go:117] "RemoveContainer" containerID="b688c3d149c912bd4bb9fcdc7fa40034120f1ce5871405625553388dc79ff017" Jul 09 23:09:45.830801 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.830781 2568 scope.go:117] "RemoveContainer" containerID="217a9cf9cdccf2b26b278e28b4d0a2f2bf09a2f05f2db82f20ff138f73d923c6" Jul 09 23:09:45.839391 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.839370 2568 scope.go:117] "RemoveContainer" containerID="b688c3d149c912bd4bb9fcdc7fa40034120f1ce5871405625553388dc79ff017" Jul 09 23:09:45.839704 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:09:45.839684 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b688c3d149c912bd4bb9fcdc7fa40034120f1ce5871405625553388dc79ff017\": container with ID starting with b688c3d149c912bd4bb9fcdc7fa40034120f1ce5871405625553388dc79ff017 not found: ID does not exist" containerID="b688c3d149c912bd4bb9fcdc7fa40034120f1ce5871405625553388dc79ff017" Jul 09 23:09:45.839749 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.839715 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b688c3d149c912bd4bb9fcdc7fa40034120f1ce5871405625553388dc79ff017"} err="failed to get container status \"b688c3d149c912bd4bb9fcdc7fa40034120f1ce5871405625553388dc79ff017\": rpc error: code = NotFound desc = could not find container \"b688c3d149c912bd4bb9fcdc7fa40034120f1ce5871405625553388dc79ff017\": container with ID starting with b688c3d149c912bd4bb9fcdc7fa40034120f1ce5871405625553388dc79ff017 not found: ID does not exist" Jul 09 23:09:45.839749 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.839734 2568 scope.go:117] "RemoveContainer" containerID="217a9cf9cdccf2b26b278e28b4d0a2f2bf09a2f05f2db82f20ff138f73d923c6" Jul 09 23:09:45.839988 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:09:45.839969 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"217a9cf9cdccf2b26b278e28b4d0a2f2bf09a2f05f2db82f20ff138f73d923c6\": container with ID starting with 217a9cf9cdccf2b26b278e28b4d0a2f2bf09a2f05f2db82f20ff138f73d923c6 not found: ID does not exist" containerID="217a9cf9cdccf2b26b278e28b4d0a2f2bf09a2f05f2db82f20ff138f73d923c6" Jul 09 23:09:45.840026 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.839997 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217a9cf9cdccf2b26b278e28b4d0a2f2bf09a2f05f2db82f20ff138f73d923c6"} err="failed to get container status \"217a9cf9cdccf2b26b278e28b4d0a2f2bf09a2f05f2db82f20ff138f73d923c6\": rpc error: code = NotFound desc = could not find container \"217a9cf9cdccf2b26b278e28b4d0a2f2bf09a2f05f2db82f20ff138f73d923c6\": container with ID starting with 217a9cf9cdccf2b26b278e28b4d0a2f2bf09a2f05f2db82f20ff138f73d923c6 not found: ID does not exist" Jul 09 23:09:45.852188 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.852156 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp"] Jul 09 23:09:45.854327 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:45.854306 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-728627-openshift-default-f4d9d675f-24fxp"] Jul 09 23:09:46.047802 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:46.047721 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6ca8b55-a523-4c15-ac8f-70a5939d41bb" path="/var/lib/kubelet/pods/e6ca8b55-a523-4c15-ac8f-70a5939d41bb/volumes" Jul 09 23:09:46.785823 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:46.785789 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:46.886506 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:46.886456 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:47.068318 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.068286 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:47.296039 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.295999 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:47.629424 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.629386 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:47.927528 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.927414 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb"] Jul 09 23:09:47.927922 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.927904 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6ca8b55-a523-4c15-ac8f-70a5939d41bb" containerName="istio-proxy" Jul 09 23:09:47.927976 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.927927 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6ca8b55-a523-4c15-ac8f-70a5939d41bb" containerName="istio-proxy" Jul 09 23:09:47.927976 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.927940 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6ca8b55-a523-4c15-ac8f-70a5939d41bb" containerName="istio-proxy" Jul 09 23:09:47.927976 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.927949 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6ca8b55-a523-4c15-ac8f-70a5939d41bb" containerName="istio-proxy" Jul 09 23:09:47.927976 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.927960 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa642935-9a53-4abd-a54a-6e616259e1b2" containerName="maas-api" Jul 09 23:09:47.927976 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.927968 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa642935-9a53-4abd-a54a-6e616259e1b2" containerName="maas-api" Jul 09 23:09:47.928132 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.927979 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25cacd18-8950-4448-b39f-7d11275241b4" containerName="maas-api" Jul 09 23:09:47.928132 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.927988 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="25cacd18-8950-4448-b39f-7d11275241b4" containerName="maas-api" Jul 09 23:09:47.928132 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.928079 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6ca8b55-a523-4c15-ac8f-70a5939d41bb" containerName="istio-proxy" Jul 09 23:09:47.928132 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.928087 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6ca8b55-a523-4c15-ac8f-70a5939d41bb" containerName="istio-proxy" Jul 09 23:09:47.928132 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.928095 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="25cacd18-8950-4448-b39f-7d11275241b4" containerName="maas-api" Jul 09 23:09:47.928132 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.928101 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa642935-9a53-4abd-a54a-6e616259e1b2" containerName="maas-api" Jul 09 23:09:47.932859 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.932836 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:47.935653 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.935625 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-566ef9-openshift-default-dockercfg-x97vq\"" Jul 09 23:09:47.940817 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.940789 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb"] Jul 09 23:09:47.948287 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:47.948255 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:48.054706 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.054677 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-workload-socket\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.054897 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.054734 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7620603b-8ed9-447c-96ac-afc67456a0ec-istiod-ca-cert\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.054897 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.054771 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-envoy\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.054897 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.054787 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-podinfo\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.054897 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.054804 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwpcp\" (UniqueName: \"kubernetes.io/projected/7620603b-8ed9-447c-96ac-afc67456a0ec-kube-api-access-mwpcp\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.054897 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.054855 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-credential-socket\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.055122 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.054934 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-data\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.055122 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.054992 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-token\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.055122 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.055017 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-workload-certs\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.156442 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.156397 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-workload-socket\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.156656 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.156458 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7620603b-8ed9-447c-96ac-afc67456a0ec-istiod-ca-cert\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.156656 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.156522 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-envoy\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.156656 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.156544 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-podinfo\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.156656 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.156569 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwpcp\" (UniqueName: \"kubernetes.io/projected/7620603b-8ed9-447c-96ac-afc67456a0ec-kube-api-access-mwpcp\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.156656 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.156599 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-credential-socket\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.156938 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.156657 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-data\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.156938 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.156720 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-token\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.156938 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.156746 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-workload-certs\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.157108 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.156989 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-workload-socket\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.157108 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.157053 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-credential-socket\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.157297 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.157266 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-workload-certs\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.157370 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.157310 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7620603b-8ed9-447c-96ac-afc67456a0ec-istiod-ca-cert\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.157370 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.157327 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-data\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.158895 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.158865 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-envoy\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.159117 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.159099 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-podinfo\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.164647 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.164618 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-token\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.164762 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.164743 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwpcp\" (UniqueName: \"kubernetes.io/projected/7620603b-8ed9-447c-96ac-afc67456a0ec-kube-api-access-mwpcp\") pod \"e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.245839 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.245745 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:48.386734 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.386702 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb"] Jul 09 23:09:48.388689 ip-10-0-135-80 kubenswrapper[2568]: W0709 23:09:48.388660 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7620603b_8ed9_447c_96ac_afc67456a0ec.slice/crio-4497e7ecd51d02685f2d28f2a91be92cbad92c282fa1a7255f6a1c782b76409d WatchSource:0}: Error finding container 4497e7ecd51d02685f2d28f2a91be92cbad92c282fa1a7255f6a1c782b76409d: Status 404 returned error can't find the container with id 4497e7ecd51d02685f2d28f2a91be92cbad92c282fa1a7255f6a1c782b76409d Jul 09 23:09:48.390994 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.390960 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:09:48.391111 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.391029 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:09:48.391111 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.391059 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:09:48.460256 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.460226 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:48.839469 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.839435 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" event={"ID":"7620603b-8ed9-447c-96ac-afc67456a0ec","Type":"ContainerStarted","Data":"c541e9688aed0bebca4000f8ee77c8764cdbecb16e80f615eb869c31f152a269"} Jul 09 23:09:48.839679 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.839508 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" event={"ID":"7620603b-8ed9-447c-96ac-afc67456a0ec","Type":"ContainerStarted","Data":"4497e7ecd51d02685f2d28f2a91be92cbad92c282fa1a7255f6a1c782b76409d"} Jul 09 23:09:48.862552 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:48.862473 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" podStartSLOduration=1.862452827 podStartE2EDuration="1.862452827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:09:48.859803812 +0000 UTC m=+2213.400882606" watchObservedRunningTime="2026-07-09 23:09:48.862452827 +0000 UTC m=+2213.403531622" Jul 09 23:09:49.246980 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:49.246883 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:49.252213 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:49.252183 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:49.845285 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:49.845252 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:49.846403 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:49.846383 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:09:50.801609 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:50.801571 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:50.918166 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:50.918119 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:50.929237 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:50.929198 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll"] Jul 09 23:09:50.932813 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:50.932793 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" Jul 09 23:09:50.935811 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:50.935790 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-566ef9-serving-cert\"" Jul 09 23:09:50.950028 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:50.950002 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll"] Jul 09 23:09:51.025684 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:51.025648 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:51.085099 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:51.085061 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dff2859-d928-4979-8d46-03edee62eb9f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll\" (UID: \"6dff2859-d928-4979-8d46-03edee62eb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" Jul 09 23:09:51.085302 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:51.085107 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89dsd\" (UniqueName: \"kubernetes.io/projected/6dff2859-d928-4979-8d46-03edee62eb9f-kube-api-access-89dsd\") pod \"maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll\" (UID: \"6dff2859-d928-4979-8d46-03edee62eb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" Jul 09 23:09:51.085302 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:51.085184 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dff2859-d928-4979-8d46-03edee62eb9f-maas-api-tls\") pod \"maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll\" (UID: \"6dff2859-d928-4979-8d46-03edee62eb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" Jul 09 23:09:51.126201 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:51.126161 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:51.186225 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:51.186184 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dff2859-d928-4979-8d46-03edee62eb9f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll\" (UID: \"6dff2859-d928-4979-8d46-03edee62eb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" Jul 09 23:09:51.186417 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:51.186236 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89dsd\" (UniqueName: \"kubernetes.io/projected/6dff2859-d928-4979-8d46-03edee62eb9f-kube-api-access-89dsd\") pod \"maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll\" (UID: \"6dff2859-d928-4979-8d46-03edee62eb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" Jul 09 23:09:51.186417 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:51.186271 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dff2859-d928-4979-8d46-03edee62eb9f-maas-api-tls\") pod \"maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll\" (UID: \"6dff2859-d928-4979-8d46-03edee62eb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" Jul 09 23:09:51.186900 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:51.186879 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dff2859-d928-4979-8d46-03edee62eb9f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll\" (UID: \"6dff2859-d928-4979-8d46-03edee62eb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" Jul 09 23:09:51.188803 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:51.188785 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dff2859-d928-4979-8d46-03edee62eb9f-maas-api-tls\") pod \"maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll\" (UID: \"6dff2859-d928-4979-8d46-03edee62eb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" Jul 09 23:09:51.197046 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:51.197018 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89dsd\" (UniqueName: \"kubernetes.io/projected/6dff2859-d928-4979-8d46-03edee62eb9f-kube-api-access-89dsd\") pod \"maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll\" (UID: \"6dff2859-d928-4979-8d46-03edee62eb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" Jul 09 23:09:51.245122 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:51.245090 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" Jul 09 23:09:51.386841 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:51.386812 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll"] Jul 09 23:09:51.388375 ip-10-0-135-80 kubenswrapper[2568]: W0709 23:09:51.388346 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dff2859_d928_4979_8d46_03edee62eb9f.slice/crio-ff2b9ad681366b69b9d211a90f5a141ae61763cd0a210e3adc4df75fbdf09803 WatchSource:0}: Error finding container ff2b9ad681366b69b9d211a90f5a141ae61763cd0a210e3adc4df75fbdf09803: Status 404 returned error can't find the container with id ff2b9ad681366b69b9d211a90f5a141ae61763cd0a210e3adc4df75fbdf09803 Jul 09 23:09:51.496178 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:51.496142 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:51.599359 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:51.599322 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:51.862802 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:51.862765 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" event={"ID":"6dff2859-d928-4979-8d46-03edee62eb9f","Type":"ContainerStarted","Data":"4544c09a5ac21faea435483a9e51d8f2011470deb49641f779eb70fc5d0f9c32"} Jul 09 23:09:51.863201 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:51.862821 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" event={"ID":"6dff2859-d928-4979-8d46-03edee62eb9f","Type":"ContainerStarted","Data":"ff2b9ad681366b69b9d211a90f5a141ae61763cd0a210e3adc4df75fbdf09803"} Jul 09 23:09:51.863201 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:51.863111 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" Jul 09 23:09:51.882079 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:51.882017 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" podStartSLOduration=1.544751236 podStartE2EDuration="1.882000224s" totalImagesPullingTime="337.248988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:09:51.880629905 +0000 UTC m=+2216.421708723" watchObservedRunningTime="2026-07-09 23:09:51.882000224 +0000 UTC m=+2216.423079017" Jul 09 23:09:52.221818 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:52.221767 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:52.320161 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:52.320132 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:53.019777 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:53.019744 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:53.120956 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:53.120916 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:55.325182 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:55.325141 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:55.675378 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:55.675288 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:55.992276 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:55.992191 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:56.180983 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:56.180944 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:57.874348 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:57.874318 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" Jul 09 23:09:59.074406 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.074360 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:59.194704 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.194667 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8"] Jul 09 23:09:59.199459 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.199437 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" Jul 09 23:09:59.203050 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.203025 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-57ed9c-serving-cert\"" Jul 09 23:09:59.210217 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.210184 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:59.212313 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.212285 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8"] Jul 09 23:09:59.308054 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.308015 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:59.367645 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.367562 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0a3c59a-9adc-4e12-bf72-4cd28f8f265a-maas-api-tls\") pod \"maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8\" (UID: \"a0a3c59a-9adc-4e12-bf72-4cd28f8f265a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" Jul 09 23:09:59.367645 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.367602 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qb4m\" (UniqueName: \"kubernetes.io/projected/a0a3c59a-9adc-4e12-bf72-4cd28f8f265a-kube-api-access-4qb4m\") pod \"maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8\" (UID: \"a0a3c59a-9adc-4e12-bf72-4cd28f8f265a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" Jul 09 23:09:59.367834 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.367762 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a3c59a-9adc-4e12-bf72-4cd28f8f265a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8\" (UID: \"a0a3c59a-9adc-4e12-bf72-4cd28f8f265a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" Jul 09 23:09:59.408342 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.408284 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:59.469200 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.469157 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a3c59a-9adc-4e12-bf72-4cd28f8f265a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8\" (UID: \"a0a3c59a-9adc-4e12-bf72-4cd28f8f265a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" Jul 09 23:09:59.469377 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.469214 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0a3c59a-9adc-4e12-bf72-4cd28f8f265a-maas-api-tls\") pod \"maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8\" (UID: \"a0a3c59a-9adc-4e12-bf72-4cd28f8f265a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" Jul 09 23:09:59.469377 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.469253 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qb4m\" (UniqueName: \"kubernetes.io/projected/a0a3c59a-9adc-4e12-bf72-4cd28f8f265a-kube-api-access-4qb4m\") pod \"maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8\" (UID: \"a0a3c59a-9adc-4e12-bf72-4cd28f8f265a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" Jul 09 23:09:59.469930 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.469912 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a3c59a-9adc-4e12-bf72-4cd28f8f265a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8\" (UID: \"a0a3c59a-9adc-4e12-bf72-4cd28f8f265a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" Jul 09 23:09:59.471833 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.471806 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0a3c59a-9adc-4e12-bf72-4cd28f8f265a-maas-api-tls\") pod \"maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8\" (UID: \"a0a3c59a-9adc-4e12-bf72-4cd28f8f265a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" Jul 09 23:09:59.478469 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.478441 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qb4m\" (UniqueName: \"kubernetes.io/projected/a0a3c59a-9adc-4e12-bf72-4cd28f8f265a-kube-api-access-4qb4m\") pod \"maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8\" (UID: \"a0a3c59a-9adc-4e12-bf72-4cd28f8f265a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" Jul 09 23:09:59.510273 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.510238 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" Jul 09 23:09:59.660714 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.660684 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8"] Jul 09 23:09:59.662051 ip-10-0-135-80 kubenswrapper[2568]: W0709 23:09:59.662022 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a3c59a_9adc_4e12_bf72_4cd28f8f265a.slice/crio-3ed87680d4be3d9d29e420b9351d3540185460b7e901d5d40d731000190a2c01 WatchSource:0}: Error finding container 3ed87680d4be3d9d29e420b9351d3540185460b7e901d5d40d731000190a2c01: Status 404 returned error can't find the container with id 3ed87680d4be3d9d29e420b9351d3540185460b7e901d5d40d731000190a2c01 Jul 09 23:09:59.791799 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.791760 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:59.897370 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.897337 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:09:59.909301 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:09:59.909208 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" event={"ID":"a0a3c59a-9adc-4e12-bf72-4cd28f8f265a","Type":"ContainerStarted","Data":"3ed87680d4be3d9d29e420b9351d3540185460b7e901d5d40d731000190a2c01"} Jul 09 23:10:00.915924 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:00.915883 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" event={"ID":"a0a3c59a-9adc-4e12-bf72-4cd28f8f265a","Type":"ContainerStarted","Data":"5479e7c53b4f8c3fcdc9c49ffccc83cac8ed038e7965f6f9a75ab9c851f21809"} Jul 09 23:10:00.916411 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:00.916055 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" Jul 09 23:10:00.932661 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:00.932602 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" podStartSLOduration=1.493971657 podStartE2EDuration="1.93258382s" totalImagesPullingTime="438.612163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:10:00.931196217 +0000 UTC m=+2225.472275057" watchObservedRunningTime="2026-07-09 23:10:00.93258382 +0000 UTC m=+2225.473662613" Jul 09 23:10:03.610376 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:03.610325 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:10:05.888625 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:05.888588 2568 patch_prober.go:28] interesting pod/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 23:10:05.889039 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:05.888641 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" podUID="7620603b-8ed9-447c-96ac-afc67456a0ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 23:10:06.927353 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:06.927259 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" Jul 09 23:10:06.952096 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:06.952063 2568 generic.go:356] "Generic (PLEG): container finished" podID="7620603b-8ed9-447c-96ac-afc67456a0ec" containerID="c541e9688aed0bebca4000f8ee77c8764cdbecb16e80f615eb869c31f152a269" exitCode=0 Jul 09 23:10:06.952263 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:06.952139 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" event={"ID":"7620603b-8ed9-447c-96ac-afc67456a0ec","Type":"ContainerDied","Data":"c541e9688aed0bebca4000f8ee77c8764cdbecb16e80f615eb869c31f152a269"} Jul 09 23:10:06.952600 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:06.952582 2568 scope.go:117] "RemoveContainer" containerID="c541e9688aed0bebca4000f8ee77c8764cdbecb16e80f615eb869c31f152a269" Jul 09 23:10:06.953747 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:06.953714 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:10:06.953864 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:06.953792 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:10:06.953864 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:06.953825 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 23:10:07.959991 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:07.959953 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" event={"ID":"7620603b-8ed9-447c-96ac-afc67456a0ec","Type":"ContainerStarted","Data":"31b413eb1c78d85ab11a0f5f1d2a6b25ec5e88fdcb5fe985d36e3b7f3003c32c"} Jul 09 23:10:08.246568 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:08.246463 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:10:08.246756 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:08.246734 2568 patch_prober.go:28] interesting pod/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 09 23:10:08.246817 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:08.246799 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" podUID="7620603b-8ed9-447c-96ac-afc67456a0ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 09 23:10:09.246668 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:09.246629 2568 patch_prober.go:28] interesting pod/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 09 23:10:09.247072 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:09.246688 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" podUID="7620603b-8ed9-447c-96ac-afc67456a0ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 09 23:10:10.246916 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:10.246874 2568 patch_prober.go:28] interesting pod/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 09 23:10:10.247335 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:10.246935 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" podUID="7620603b-8ed9-447c-96ac-afc67456a0ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 09 23:10:11.246423 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:11.246385 2568 patch_prober.go:28] interesting pod/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 09 23:10:11.246639 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:11.246442 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" podUID="7620603b-8ed9-447c-96ac-afc67456a0ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 09 23:10:12.206826 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:12.206789 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:10:12.249859 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:12.249828 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:10:12.250244 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:12.250223 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:10:12.251061 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:12.251042 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:10:12.277287 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:12.272721 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:10:20.602668 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:20.602575 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:10:21.474740 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:21.474705 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:10:21.586400 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:21.586365 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:10:50.711620 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:10:50.711547 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:11:31.673580 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:11:31.673539 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:11:48.374278 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:11:48.374186 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:12:14.675110 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:12:14.675074 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:12:25.506989 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:12:25.506929 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:12:26.333756 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:12:26.333718 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:12:58.764098 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:12:58.764062 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:12:59.610126 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:12:59.610092 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:13:03.205723 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:13:03.205684 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:13:18.101044 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:13:18.100963 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:13:26.666343 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:13:26.666302 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:13:26.767153 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:13:26.767124 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:14:05.630336 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:14:05.630296 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:14:06.536620 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:14:06.536583 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:14:09.663257 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:14:09.663221 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:14:50.203618 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:14:50.203529 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:14:53.005984 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:14:53.005943 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:14:53.107356 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:14:53.107321 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:15:00.144129 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:00.144096 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-566ef9-29727315-8cplq"] Jul 09 23:15:00.147830 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:00.147813 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-566ef9-29727315-8cplq" Jul 09 23:15:00.150543 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:00.150519 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hn6dj\"" Jul 09 23:15:00.154385 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:00.154200 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-566ef9-29727315-8cplq"] Jul 09 23:15:00.159446 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:00.159428 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-566ef9-29727315-8cplq" Jul 09 23:15:00.296417 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:00.296388 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-566ef9-29727315-8cplq"] Jul 09 23:15:00.298178 ip-10-0-135-80 kubenswrapper[2568]: W0709 23:15:00.298152 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a5e362_046e_472c_9385_938de141eacd.slice/crio-e0659c2b363bcf5253bd098e5b4e74caa98e1cbc49d06ec4c0c404f5caf48402 WatchSource:0}: Error finding container e0659c2b363bcf5253bd098e5b4e74caa98e1cbc49d06ec4c0c404f5caf48402: Status 404 returned error can't find the container with id e0659c2b363bcf5253bd098e5b4e74caa98e1cbc49d06ec4c0c404f5caf48402 Jul 09 23:15:00.526635 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:00.526600 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-566ef9-29727315-8cplq" event={"ID":"78a5e362-046e-472c-9385-938de141eacd","Type":"ContainerStarted","Data":"e0659c2b363bcf5253bd098e5b4e74caa98e1cbc49d06ec4c0c404f5caf48402"} Jul 09 23:15:01.532963 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:01.532925 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-566ef9-29727315-8cplq" event={"ID":"78a5e362-046e-472c-9385-938de141eacd","Type":"ContainerStarted","Data":"4ee06f330ab4a3f0304410744325884136132320ff69bc508bb184845a7db75a"} Jul 09 23:15:01.549341 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:01.549283 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-566ef9-29727315-8cplq" podStartSLOduration=1.549266824 podStartE2EDuration="1.549266824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:15:01.546563785 +0000 UTC m=+2526.087642578" watchObservedRunningTime="2026-07-09 23:15:01.549266824 +0000 UTC m=+2526.090345619" Jul 09 23:15:10.443752 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:10.443711 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:15:13.005375 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:13.005340 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:15:13.107924 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:13.107893 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:15:22.249376 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:22.249336 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:15:22.350019 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:22.349989 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:15:38.296294 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:38.296263 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:15:39.176091 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:39.176048 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:15:58.329196 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:58.329153 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:15:58.812906 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:58.812870 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:15:58.914704 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:58.914667 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:15:59.014052 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.014021 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:15:59.108403 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.108321 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll"] Jul 09 23:15:59.108593 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.108560 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" podUID="6dff2859-d928-4979-8d46-03edee62eb9f" containerName="maas-api" containerID="cri-o://4544c09a5ac21faea435483a9e51d8f2011470deb49641f779eb70fc5d0f9c32" gracePeriod=30 Jul 09 23:15:59.132920 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.132884 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:15:59.142364 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.142336 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-566ef9-29727315-8cplq"] Jul 09 23:15:59.142569 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.142546 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-566ef9-29727315-8cplq" podUID="78a5e362-046e-472c-9385-938de141eacd" containerName="cleanup" containerID="cri-o://4ee06f330ab4a3f0304410744325884136132320ff69bc508bb184845a7db75a" gracePeriod=30 Jul 09 23:15:59.554157 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.554132 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" Jul 09 23:15:59.654564 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.654437 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dff2859-d928-4979-8d46-03edee62eb9f-maas-api-tls\") pod \"6dff2859-d928-4979-8d46-03edee62eb9f\" (UID: \"6dff2859-d928-4979-8d46-03edee62eb9f\") " Jul 09 23:15:59.654564 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.654550 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89dsd\" (UniqueName: \"kubernetes.io/projected/6dff2859-d928-4979-8d46-03edee62eb9f-kube-api-access-89dsd\") pod \"6dff2859-d928-4979-8d46-03edee62eb9f\" (UID: \"6dff2859-d928-4979-8d46-03edee62eb9f\") " Jul 09 23:15:59.654825 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.654589 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dff2859-d928-4979-8d46-03edee62eb9f-openshift-service-ca\") pod \"6dff2859-d928-4979-8d46-03edee62eb9f\" (UID: \"6dff2859-d928-4979-8d46-03edee62eb9f\") " Jul 09 23:15:59.654999 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.654974 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dff2859-d928-4979-8d46-03edee62eb9f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6dff2859-d928-4979-8d46-03edee62eb9f" (UID: "6dff2859-d928-4979-8d46-03edee62eb9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:15:59.656562 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.656530 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dff2859-d928-4979-8d46-03edee62eb9f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6dff2859-d928-4979-8d46-03edee62eb9f" (UID: "6dff2859-d928-4979-8d46-03edee62eb9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:15:59.656657 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.656580 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dff2859-d928-4979-8d46-03edee62eb9f-kube-api-access-89dsd" (OuterVolumeSpecName: "kube-api-access-89dsd") pod "6dff2859-d928-4979-8d46-03edee62eb9f" (UID: "6dff2859-d928-4979-8d46-03edee62eb9f"). InnerVolumeSpecName "kube-api-access-89dsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:15:59.755448 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.755409 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89dsd\" (UniqueName: \"kubernetes.io/projected/6dff2859-d928-4979-8d46-03edee62eb9f-kube-api-access-89dsd\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:15:59.755448 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.755439 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dff2859-d928-4979-8d46-03edee62eb9f-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:15:59.755448 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.755449 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dff2859-d928-4979-8d46-03edee62eb9f-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:15:59.842060 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.842020 2568 generic.go:356] "Generic (PLEG): container finished" podID="6dff2859-d928-4979-8d46-03edee62eb9f" containerID="4544c09a5ac21faea435483a9e51d8f2011470deb49641f779eb70fc5d0f9c32" exitCode=0 Jul 09 23:15:59.842223 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.842073 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" Jul 09 23:15:59.842223 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.842110 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" event={"ID":"6dff2859-d928-4979-8d46-03edee62eb9f","Type":"ContainerDied","Data":"4544c09a5ac21faea435483a9e51d8f2011470deb49641f779eb70fc5d0f9c32"} Jul 09 23:15:59.842223 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.842172 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll" event={"ID":"6dff2859-d928-4979-8d46-03edee62eb9f","Type":"ContainerDied","Data":"ff2b9ad681366b69b9d211a90f5a141ae61763cd0a210e3adc4df75fbdf09803"} Jul 09 23:15:59.842223 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.842196 2568 scope.go:117] "RemoveContainer" containerID="4544c09a5ac21faea435483a9e51d8f2011470deb49641f779eb70fc5d0f9c32" Jul 09 23:15:59.851586 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.851562 2568 scope.go:117] "RemoveContainer" containerID="4544c09a5ac21faea435483a9e51d8f2011470deb49641f779eb70fc5d0f9c32" Jul 09 23:15:59.851855 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:15:59.851836 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4544c09a5ac21faea435483a9e51d8f2011470deb49641f779eb70fc5d0f9c32\": container with ID starting with 4544c09a5ac21faea435483a9e51d8f2011470deb49641f779eb70fc5d0f9c32 not found: ID does not exist" containerID="4544c09a5ac21faea435483a9e51d8f2011470deb49641f779eb70fc5d0f9c32" Jul 09 23:15:59.851898 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.851864 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4544c09a5ac21faea435483a9e51d8f2011470deb49641f779eb70fc5d0f9c32"} err="failed to get container status \"4544c09a5ac21faea435483a9e51d8f2011470deb49641f779eb70fc5d0f9c32\": rpc error: code = NotFound desc = could not find container \"4544c09a5ac21faea435483a9e51d8f2011470deb49641f779eb70fc5d0f9c32\": container with ID starting with 4544c09a5ac21faea435483a9e51d8f2011470deb49641f779eb70fc5d0f9c32 not found: ID does not exist" Jul 09 23:15:59.864143 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.864077 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll"] Jul 09 23:15:59.866403 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:15:59.866381 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-566ef9-7f46c6d88d-48rll"] Jul 09 23:16:00.048834 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:00.048741 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dff2859-d928-4979-8d46-03edee62eb9f" path="/var/lib/kubelet/pods/6dff2859-d928-4979-8d46-03edee62eb9f/volumes" Jul 09 23:16:04.570297 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:04.570257 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:16:04.630006 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:04.629971 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:16:04.821216 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:04.821111 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb"] Jul 09 23:16:04.821416 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:04.821383 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" podUID="7620603b-8ed9-447c-96ac-afc67456a0ec" containerName="istio-proxy" containerID="cri-o://31b413eb1c78d85ab11a0f5f1d2a6b25ec5e88fdcb5fe985d36e3b7f3003c32c" gracePeriod=30 Jul 09 23:16:04.830448 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:04.830420 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:16:05.073262 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:05.073159 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:16:05.238958 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:05.238917 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:16:05.682671 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:05.682634 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:16:05.784250 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:05.784213 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:16:05.880736 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:05.880701 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:16:05.927774 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:05.927742 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8"] Jul 09 23:16:05.927992 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:05.927958 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" podUID="a0a3c59a-9adc-4e12-bf72-4cd28f8f265a" containerName="maas-api" containerID="cri-o://5479e7c53b4f8c3fcdc9c49ffccc83cac8ed038e7965f6f9a75ab9c851f21809" gracePeriod=30 Jul 09 23:16:05.981422 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:05.981321 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:16:06.378681 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:06.378656 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" Jul 09 23:16:06.419669 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:06.419637 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0a3c59a-9adc-4e12-bf72-4cd28f8f265a-maas-api-tls\") pod \"a0a3c59a-9adc-4e12-bf72-4cd28f8f265a\" (UID: \"a0a3c59a-9adc-4e12-bf72-4cd28f8f265a\") " Jul 09 23:16:06.419876 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:06.419693 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a3c59a-9adc-4e12-bf72-4cd28f8f265a-openshift-service-ca\") pod \"a0a3c59a-9adc-4e12-bf72-4cd28f8f265a\" (UID: \"a0a3c59a-9adc-4e12-bf72-4cd28f8f265a\") " Jul 09 23:16:06.419876 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:06.419775 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qb4m\" (UniqueName: \"kubernetes.io/projected/a0a3c59a-9adc-4e12-bf72-4cd28f8f265a-kube-api-access-4qb4m\") pod \"a0a3c59a-9adc-4e12-bf72-4cd28f8f265a\" (UID: \"a0a3c59a-9adc-4e12-bf72-4cd28f8f265a\") " Jul 09 23:16:06.420132 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:06.420108 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a3c59a-9adc-4e12-bf72-4cd28f8f265a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a0a3c59a-9adc-4e12-bf72-4cd28f8f265a" (UID: "a0a3c59a-9adc-4e12-bf72-4cd28f8f265a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:16:06.421853 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:06.421827 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0a3c59a-9adc-4e12-bf72-4cd28f8f265a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a0a3c59a-9adc-4e12-bf72-4cd28f8f265a" (UID: "a0a3c59a-9adc-4e12-bf72-4cd28f8f265a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:16:06.421952 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:06.421877 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a3c59a-9adc-4e12-bf72-4cd28f8f265a-kube-api-access-4qb4m" (OuterVolumeSpecName: "kube-api-access-4qb4m") pod "a0a3c59a-9adc-4e12-bf72-4cd28f8f265a" (UID: "a0a3c59a-9adc-4e12-bf72-4cd28f8f265a"). InnerVolumeSpecName "kube-api-access-4qb4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:16:06.521288 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:06.521204 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0a3c59a-9adc-4e12-bf72-4cd28f8f265a-maas-api-tls\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:16:06.521288 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:06.521232 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a3c59a-9adc-4e12-bf72-4cd28f8f265a-openshift-service-ca\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:16:06.521288 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:06.521242 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qb4m\" (UniqueName: \"kubernetes.io/projected/a0a3c59a-9adc-4e12-bf72-4cd28f8f265a-kube-api-access-4qb4m\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:16:06.887367 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:06.887330 2568 generic.go:356] "Generic (PLEG): container finished" podID="a0a3c59a-9adc-4e12-bf72-4cd28f8f265a" containerID="5479e7c53b4f8c3fcdc9c49ffccc83cac8ed038e7965f6f9a75ab9c851f21809" exitCode=0 Jul 09 23:16:06.887367 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:06.887370 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" Jul 09 23:16:06.887928 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:06.887391 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" event={"ID":"a0a3c59a-9adc-4e12-bf72-4cd28f8f265a","Type":"ContainerDied","Data":"5479e7c53b4f8c3fcdc9c49ffccc83cac8ed038e7965f6f9a75ab9c851f21809"} Jul 09 23:16:06.887928 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:06.887447 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8" event={"ID":"a0a3c59a-9adc-4e12-bf72-4cd28f8f265a","Type":"ContainerDied","Data":"3ed87680d4be3d9d29e420b9351d3540185460b7e901d5d40d731000190a2c01"} Jul 09 23:16:06.887928 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:06.887468 2568 scope.go:117] "RemoveContainer" containerID="5479e7c53b4f8c3fcdc9c49ffccc83cac8ed038e7965f6f9a75ab9c851f21809" Jul 09 23:16:06.897058 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:06.897039 2568 scope.go:117] "RemoveContainer" containerID="5479e7c53b4f8c3fcdc9c49ffccc83cac8ed038e7965f6f9a75ab9c851f21809" Jul 09 23:16:06.897343 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:16:06.897323 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5479e7c53b4f8c3fcdc9c49ffccc83cac8ed038e7965f6f9a75ab9c851f21809\": container with ID starting with 5479e7c53b4f8c3fcdc9c49ffccc83cac8ed038e7965f6f9a75ab9c851f21809 not found: ID does not exist" containerID="5479e7c53b4f8c3fcdc9c49ffccc83cac8ed038e7965f6f9a75ab9c851f21809" Jul 09 23:16:06.897407 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:06.897352 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5479e7c53b4f8c3fcdc9c49ffccc83cac8ed038e7965f6f9a75ab9c851f21809"} err="failed to get container status \"5479e7c53b4f8c3fcdc9c49ffccc83cac8ed038e7965f6f9a75ab9c851f21809\": rpc error: code = NotFound desc = could not find container \"5479e7c53b4f8c3fcdc9c49ffccc83cac8ed038e7965f6f9a75ab9c851f21809\": container with ID starting with 5479e7c53b4f8c3fcdc9c49ffccc83cac8ed038e7965f6f9a75ab9c851f21809 not found: ID does not exist" Jul 09 23:16:06.909261 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:06.909229 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8"] Jul 09 23:16:06.911279 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:06.911244 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-57ed9c-76f86d956c-48wg8"] Jul 09 23:16:08.048268 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:08.048225 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0a3c59a-9adc-4e12-bf72-4cd28f8f265a" path="/var/lib/kubelet/pods/a0a3c59a-9adc-4e12-bf72-4cd28f8f265a/volumes" Jul 09 23:16:10.080632 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.080558 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:16:10.152742 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.152700 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-credential-socket\") pod \"7620603b-8ed9-447c-96ac-afc67456a0ec\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " Jul 09 23:16:10.152935 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.152758 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-envoy\") pod \"7620603b-8ed9-447c-96ac-afc67456a0ec\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " Jul 09 23:16:10.152935 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.152789 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-workload-socket\") pod \"7620603b-8ed9-447c-96ac-afc67456a0ec\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " Jul 09 23:16:10.152935 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.152805 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7620603b-8ed9-447c-96ac-afc67456a0ec-istiod-ca-cert\") pod \"7620603b-8ed9-447c-96ac-afc67456a0ec\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " Jul 09 23:16:10.152935 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.152841 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-workload-certs\") pod \"7620603b-8ed9-447c-96ac-afc67456a0ec\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " Jul 09 23:16:10.152935 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.152864 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-podinfo\") pod \"7620603b-8ed9-447c-96ac-afc67456a0ec\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " Jul 09 23:16:10.152935 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.152892 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwpcp\" (UniqueName: \"kubernetes.io/projected/7620603b-8ed9-447c-96ac-afc67456a0ec-kube-api-access-mwpcp\") pod \"7620603b-8ed9-447c-96ac-afc67456a0ec\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " Jul 09 23:16:10.153270 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.152941 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-data\") pod \"7620603b-8ed9-447c-96ac-afc67456a0ec\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " Jul 09 23:16:10.153270 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.153048 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7620603b-8ed9-447c-96ac-afc67456a0ec" (UID: "7620603b-8ed9-447c-96ac-afc67456a0ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:16:10.153270 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.153113 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7620603b-8ed9-447c-96ac-afc67456a0ec" (UID: "7620603b-8ed9-447c-96ac-afc67456a0ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:16:10.153270 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.153126 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-token\") pod \"7620603b-8ed9-447c-96ac-afc67456a0ec\" (UID: \"7620603b-8ed9-447c-96ac-afc67456a0ec\") " Jul 09 23:16:10.153270 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.153218 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7620603b-8ed9-447c-96ac-afc67456a0ec" (UID: "7620603b-8ed9-447c-96ac-afc67456a0ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:16:10.153534 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.153375 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7620603b-8ed9-447c-96ac-afc67456a0ec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7620603b-8ed9-447c-96ac-afc67456a0ec" (UID: "7620603b-8ed9-447c-96ac-afc67456a0ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:16:10.153534 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.153507 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-workload-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:16:10.153534 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.153526 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7620603b-8ed9-447c-96ac-afc67456a0ec-istiod-ca-cert\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:16:10.153694 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.153539 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-workload-certs\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:16:10.153694 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.153552 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-credential-socket\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:16:10.153927 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.153906 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-data" (OuterVolumeSpecName: "istio-data") pod "7620603b-8ed9-447c-96ac-afc67456a0ec" (UID: "7620603b-8ed9-447c-96ac-afc67456a0ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:16:10.155261 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.155222 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7620603b-8ed9-447c-96ac-afc67456a0ec" (UID: "7620603b-8ed9-447c-96ac-afc67456a0ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 23:16:10.155389 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.155270 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-token" (OuterVolumeSpecName: "istio-token") pod "7620603b-8ed9-447c-96ac-afc67456a0ec" (UID: "7620603b-8ed9-447c-96ac-afc67456a0ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:16:10.155459 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.155444 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7620603b-8ed9-447c-96ac-afc67456a0ec" (UID: "7620603b-8ed9-447c-96ac-afc67456a0ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:16:10.155571 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.155544 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7620603b-8ed9-447c-96ac-afc67456a0ec-kube-api-access-mwpcp" (OuterVolumeSpecName: "kube-api-access-mwpcp") pod "7620603b-8ed9-447c-96ac-afc67456a0ec" (UID: "7620603b-8ed9-447c-96ac-afc67456a0ec"). InnerVolumeSpecName "kube-api-access-mwpcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:16:10.254759 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.254719 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-podinfo\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:16:10.254759 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.254753 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwpcp\" (UniqueName: \"kubernetes.io/projected/7620603b-8ed9-447c-96ac-afc67456a0ec-kube-api-access-mwpcp\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:16:10.254759 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.254764 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-data\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:16:10.255006 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.254774 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-token\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:16:10.255006 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.254782 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7620603b-8ed9-447c-96ac-afc67456a0ec-istio-envoy\") on node \"ip-10-0-135-80.ec2.internal\" DevicePath \"\"" Jul 09 23:16:10.915931 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.915900 2568 generic.go:356] "Generic (PLEG): container finished" podID="7620603b-8ed9-447c-96ac-afc67456a0ec" containerID="31b413eb1c78d85ab11a0f5f1d2a6b25ec5e88fdcb5fe985d36e3b7f3003c32c" exitCode=0 Jul 09 23:16:10.916098 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.915950 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" Jul 09 23:16:10.916098 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.915985 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" event={"ID":"7620603b-8ed9-447c-96ac-afc67456a0ec","Type":"ContainerDied","Data":"31b413eb1c78d85ab11a0f5f1d2a6b25ec5e88fdcb5fe985d36e3b7f3003c32c"} Jul 09 23:16:10.916098 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.916037 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb" event={"ID":"7620603b-8ed9-447c-96ac-afc67456a0ec","Type":"ContainerDied","Data":"4497e7ecd51d02685f2d28f2a91be92cbad92c282fa1a7255f6a1c782b76409d"} Jul 09 23:16:10.916098 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.916059 2568 scope.go:117] "RemoveContainer" containerID="31b413eb1c78d85ab11a0f5f1d2a6b25ec5e88fdcb5fe985d36e3b7f3003c32c" Jul 09 23:16:10.925410 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.925387 2568 scope.go:117] "RemoveContainer" containerID="c541e9688aed0bebca4000f8ee77c8764cdbecb16e80f615eb869c31f152a269" Jul 09 23:16:10.933947 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.933927 2568 scope.go:117] "RemoveContainer" containerID="31b413eb1c78d85ab11a0f5f1d2a6b25ec5e88fdcb5fe985d36e3b7f3003c32c" Jul 09 23:16:10.934280 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:16:10.934256 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b413eb1c78d85ab11a0f5f1d2a6b25ec5e88fdcb5fe985d36e3b7f3003c32c\": container with ID starting with 31b413eb1c78d85ab11a0f5f1d2a6b25ec5e88fdcb5fe985d36e3b7f3003c32c not found: ID does not exist" containerID="31b413eb1c78d85ab11a0f5f1d2a6b25ec5e88fdcb5fe985d36e3b7f3003c32c" Jul 09 23:16:10.934342 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.934294 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b413eb1c78d85ab11a0f5f1d2a6b25ec5e88fdcb5fe985d36e3b7f3003c32c"} err="failed to get container status \"31b413eb1c78d85ab11a0f5f1d2a6b25ec5e88fdcb5fe985d36e3b7f3003c32c\": rpc error: code = NotFound desc = could not find container \"31b413eb1c78d85ab11a0f5f1d2a6b25ec5e88fdcb5fe985d36e3b7f3003c32c\": container with ID starting with 31b413eb1c78d85ab11a0f5f1d2a6b25ec5e88fdcb5fe985d36e3b7f3003c32c not found: ID does not exist" Jul 09 23:16:10.934342 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.934314 2568 scope.go:117] "RemoveContainer" containerID="c541e9688aed0bebca4000f8ee77c8764cdbecb16e80f615eb869c31f152a269" Jul 09 23:16:10.934550 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:16:10.934534 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c541e9688aed0bebca4000f8ee77c8764cdbecb16e80f615eb869c31f152a269\": container with ID starting with c541e9688aed0bebca4000f8ee77c8764cdbecb16e80f615eb869c31f152a269 not found: ID does not exist" containerID="c541e9688aed0bebca4000f8ee77c8764cdbecb16e80f615eb869c31f152a269" Jul 09 23:16:10.934590 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.934555 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c541e9688aed0bebca4000f8ee77c8764cdbecb16e80f615eb869c31f152a269"} err="failed to get container status \"c541e9688aed0bebca4000f8ee77c8764cdbecb16e80f615eb869c31f152a269\": rpc error: code = NotFound desc = could not find container \"c541e9688aed0bebca4000f8ee77c8764cdbecb16e80f615eb869c31f152a269\": container with ID starting with c541e9688aed0bebca4000f8ee77c8764cdbecb16e80f615eb869c31f152a269 not found: ID does not exist" Jul 09 23:16:10.938336 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.938307 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb"] Jul 09 23:16:10.942232 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:10.942204 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-566ef9-openshift-default-78977bc967-jsxjb"] Jul 09 23:16:11.604917 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:11.604885 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:16:11.705096 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:11.705062 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:16:11.885196 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:11.885107 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:16:12.048510 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:12.048469 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7620603b-8ed9-447c-96ac-afc67456a0ec" path="/var/lib/kubelet/pods/7620603b-8ed9-447c-96ac-afc67456a0ec/volumes" Jul 09 23:16:12.125315 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:12.125281 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:16:15.575576 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:15.575539 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:16:15.631770 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:15.631736 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:16:17.811033 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:17.810951 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f9f7f5566-m4mbf_f21b9db1-27c9-4415-83f6-fd1c8be98acd/manager/0.log" Jul 09 23:16:18.454580 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:18.454554 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54dbdc554f-rk8z8_cdae79af-b625-4446-ae94-0b79e35df8fa/copy-manifests/0.log" Jul 09 23:16:18.462118 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:18.462092 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54dbdc554f-rk8z8_cdae79af-b625-4446-ae94-0b79e35df8fa/manager/0.log" Jul 09 23:16:18.827268 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:18.827240 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7b54d47c47-crcpg_51fa8f1c-8729-42f5-b123-12fc4d7909a9/maas-api/0.log" Jul 09 23:16:19.073880 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:19.073849 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-a-566ef9-29727315-8cplq_78a5e362-046e-472c-9385-938de141eacd/cleanup/0.log" Jul 09 23:16:20.133341 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:20.133311 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4_3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7/util/0.log" Jul 09 23:16:20.139413 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:20.139383 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4_3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7/pull/0.log" Jul 09 23:16:20.145438 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:20.145414 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4_3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7/extract/0.log" Jul 09 23:16:20.265855 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:20.265818 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k_b696f428-8899-4dfb-91ee-4ecf429ba33a/extract/0.log" Jul 09 23:16:20.273248 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:20.273214 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k_b696f428-8899-4dfb-91ee-4ecf429ba33a/util/0.log" Jul 09 23:16:20.280220 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:20.280196 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k_b696f428-8899-4dfb-91ee-4ecf429ba33a/pull/0.log" Jul 09 23:16:20.399861 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:20.399781 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb_f856756d-c7e6-4102-8598-cdcf4cfb62fe/util/0.log" Jul 09 23:16:20.406832 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:20.406806 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb_f856756d-c7e6-4102-8598-cdcf4cfb62fe/pull/0.log" Jul 09 23:16:20.413207 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:20.413185 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb_f856756d-c7e6-4102-8598-cdcf4cfb62fe/extract/0.log" Jul 09 23:16:20.533719 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:20.533692 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79_3c0737b8-848f-4df6-b244-8cf6cd35cb35/util/0.log" Jul 09 23:16:20.539980 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:20.539957 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79_3c0737b8-848f-4df6-b244-8cf6cd35cb35/pull/0.log" Jul 09 23:16:20.546596 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:20.546571 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79_3c0737b8-848f-4df6-b244-8cf6cd35cb35/extract/0.log" Jul 09 23:16:21.905084 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:21.905054 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4lz5v8_c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f/istio-proxy/0.log" Jul 09 23:16:22.025109 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:22.025079 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-j48wx_c49e1048-733d-41a3-b184-48173c9132bb/discovery/0.log" Jul 09 23:16:23.240563 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:23.240536 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-pc2jw_cee10461-24fb-4eb3-8289-4ab34c9cbe9e/main/0.log" Jul 09 23:16:23.479137 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:23.479105 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db72s84_12282b71-4b77-497a-abac-9042f10d922c/main/0.log" Jul 09 23:16:28.363620 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:28.363578 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:16:28.423251 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:28.423212 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" limit=1024 Jul 09 23:16:29.341754 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:29.341723 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-566ef9-29727315-8cplq" Jul 09 23:16:30.019077 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:30.019042 2568 generic.go:356] "Generic (PLEG): container finished" podID="78a5e362-046e-472c-9385-938de141eacd" containerID="4ee06f330ab4a3f0304410744325884136132320ff69bc508bb184845a7db75a" exitCode=137 Jul 09 23:16:30.019584 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:30.019097 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-566ef9-29727315-8cplq" Jul 09 23:16:30.019584 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:30.019132 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-566ef9-29727315-8cplq" event={"ID":"78a5e362-046e-472c-9385-938de141eacd","Type":"ContainerDied","Data":"4ee06f330ab4a3f0304410744325884136132320ff69bc508bb184845a7db75a"} Jul 09 23:16:30.019584 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:30.019213 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-566ef9-29727315-8cplq" event={"ID":"78a5e362-046e-472c-9385-938de141eacd","Type":"ContainerDied","Data":"e0659c2b363bcf5253bd098e5b4e74caa98e1cbc49d06ec4c0c404f5caf48402"} Jul 09 23:16:30.019584 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:30.019236 2568 scope.go:117] "RemoveContainer" containerID="4ee06f330ab4a3f0304410744325884136132320ff69bc508bb184845a7db75a" Jul 09 23:16:30.028415 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:30.028392 2568 scope.go:117] "RemoveContainer" containerID="4ee06f330ab4a3f0304410744325884136132320ff69bc508bb184845a7db75a" Jul 09 23:16:30.028741 ip-10-0-135-80 kubenswrapper[2568]: E0709 23:16:30.028723 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee06f330ab4a3f0304410744325884136132320ff69bc508bb184845a7db75a\": container with ID starting with 4ee06f330ab4a3f0304410744325884136132320ff69bc508bb184845a7db75a not found: ID does not exist" containerID="4ee06f330ab4a3f0304410744325884136132320ff69bc508bb184845a7db75a" Jul 09 23:16:30.028801 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:30.028750 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee06f330ab4a3f0304410744325884136132320ff69bc508bb184845a7db75a"} err="failed to get container status \"4ee06f330ab4a3f0304410744325884136132320ff69bc508bb184845a7db75a\": rpc error: code = NotFound desc = could not find container \"4ee06f330ab4a3f0304410744325884136132320ff69bc508bb184845a7db75a\": container with ID starting with 4ee06f330ab4a3f0304410744325884136132320ff69bc508bb184845a7db75a not found: ID does not exist" Jul 09 23:16:30.040368 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:30.040331 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-566ef9-29727315-8cplq"] Jul 09 23:16:30.047765 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:30.047737 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-566ef9-29727315-8cplq"] Jul 09 23:16:31.370835 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:31.370804 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9nmjj_44657019-d54e-4ff7-8677-c4ec3d34e1f8/global-pull-secret-syncer/0.log" Jul 09 23:16:31.488262 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:31.488230 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6vrvs_5eb6836b-6d35-49c3-899b-d9e81847d4fb/konnectivity-agent/0.log" Jul 09 23:16:31.588339 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:31.588310 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-80.ec2.internal_4bd33866a0efa68864d83a51f0f716fd/haproxy/0.log" Jul 09 23:16:32.048972 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:32.048941 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78a5e362-046e-472c-9385-938de141eacd" path="/var/lib/kubelet/pods/78a5e362-046e-472c-9385-938de141eacd/volumes" Jul 09 23:16:36.033068 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:36.033036 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4_3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7/extract/0.log" Jul 09 23:16:36.052972 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:36.052941 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4_3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7/util/0.log" Jul 09 23:16:36.074091 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:36.074059 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599fwm4_3189d1d5-f2db-4cf3-9ef2-f098a0e6a9d7/pull/0.log" Jul 09 23:16:36.101375 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:36.101349 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k_b696f428-8899-4dfb-91ee-4ecf429ba33a/extract/0.log" Jul 09 23:16:36.121476 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:36.121447 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k_b696f428-8899-4dfb-91ee-4ecf429ba33a/util/0.log" Jul 09 23:16:36.142673 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:36.142645 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06m27k_b696f428-8899-4dfb-91ee-4ecf429ba33a/pull/0.log" Jul 09 23:16:36.176560 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:36.176530 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb_f856756d-c7e6-4102-8598-cdcf4cfb62fe/extract/0.log" Jul 09 23:16:36.196578 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:36.196550 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb_f856756d-c7e6-4102-8598-cdcf4cfb62fe/util/0.log" Jul 09 23:16:36.220958 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:36.220931 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73bxrfb_f856756d-c7e6-4102-8598-cdcf4cfb62fe/pull/0.log" Jul 09 23:16:36.269925 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:36.269896 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79_3c0737b8-848f-4df6-b244-8cf6cd35cb35/extract/0.log" Jul 09 23:16:36.292805 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:36.292715 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79_3c0737b8-848f-4df6-b244-8cf6cd35cb35/util/0.log" Jul 09 23:16:36.338157 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:36.338129 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1njc79_3c0737b8-848f-4df6-b244-8cf6cd35cb35/pull/0.log" Jul 09 23:16:38.652481 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:38.652454 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_273c3606-e9c2-49da-b58d-a26c08314372/alertmanager/0.log" Jul 09 23:16:38.673716 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:38.673688 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_273c3606-e9c2-49da-b58d-a26c08314372/config-reloader/0.log" Jul 09 23:16:38.694281 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:38.694255 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_273c3606-e9c2-49da-b58d-a26c08314372/kube-rbac-proxy-web/0.log" Jul 09 23:16:38.715992 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:38.715958 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_273c3606-e9c2-49da-b58d-a26c08314372/kube-rbac-proxy/0.log" Jul 09 23:16:38.738040 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:38.738012 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_273c3606-e9c2-49da-b58d-a26c08314372/kube-rbac-proxy-metric/0.log" Jul 09 23:16:38.759257 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:38.759229 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_273c3606-e9c2-49da-b58d-a26c08314372/prom-label-proxy/0.log" Jul 09 23:16:38.783830 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:38.783798 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_273c3606-e9c2-49da-b58d-a26c08314372/init-config-reloader/0.log" Jul 09 23:16:38.854285 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:38.854253 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-c6jvr_ccfb6a57-528d-4548-9b47-571f9b97f80a/kube-state-metrics/0.log" Jul 09 23:16:38.874739 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:38.874708 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-c6jvr_ccfb6a57-528d-4548-9b47-571f9b97f80a/kube-rbac-proxy-main/0.log" Jul 09 23:16:38.893878 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:38.893849 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-c6jvr_ccfb6a57-528d-4548-9b47-571f9b97f80a/kube-rbac-proxy-self/0.log" Jul 09 23:16:38.981714 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:38.981633 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dw4vg_650070d8-38ef-4bd2-a9de-6eee8e5fff60/node-exporter/0.log" Jul 09 23:16:39.004078 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:39.004049 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dw4vg_650070d8-38ef-4bd2-a9de-6eee8e5fff60/kube-rbac-proxy/0.log" Jul 09 23:16:39.028787 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:39.028761 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dw4vg_650070d8-38ef-4bd2-a9de-6eee8e5fff60/init-textfile/0.log" Jul 09 23:16:39.496888 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:39.496859 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-tj8f6_43700bb9-ff9c-44c6-b7c4-5f83e5bb8d56/prometheus-operator-admission-webhook/0.log" Jul 09 23:16:39.526360 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:39.526333 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7597d88677-qmmsm_a2e5ee69-00e6-4411-9b81-fece72803fb2/telemeter-client/0.log" Jul 09 23:16:39.547371 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:39.547343 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7597d88677-qmmsm_a2e5ee69-00e6-4411-9b81-fece72803fb2/reload/0.log" Jul 09 23:16:39.569397 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:39.569362 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7597d88677-qmmsm_a2e5ee69-00e6-4411-9b81-fece72803fb2/kube-rbac-proxy/0.log" Jul 09 23:16:41.361690 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.361652 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f"] Jul 09 23:16:41.362202 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.362149 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7620603b-8ed9-447c-96ac-afc67456a0ec" containerName="istio-proxy" Jul 09 23:16:41.362202 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.362171 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7620603b-8ed9-447c-96ac-afc67456a0ec" containerName="istio-proxy" Jul 09 23:16:41.362202 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.362189 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dff2859-d928-4979-8d46-03edee62eb9f" containerName="maas-api" Jul 09 23:16:41.362202 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.362197 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dff2859-d928-4979-8d46-03edee62eb9f" containerName="maas-api" Jul 09 23:16:41.362418 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.362226 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a3c59a-9adc-4e12-bf72-4cd28f8f265a" containerName="maas-api" Jul 09 23:16:41.362418 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.362237 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a3c59a-9adc-4e12-bf72-4cd28f8f265a" containerName="maas-api" Jul 09 23:16:41.362418 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.362248 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78a5e362-046e-472c-9385-938de141eacd" containerName="cleanup" Jul 09 23:16:41.362418 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.362255 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="78a5e362-046e-472c-9385-938de141eacd" containerName="cleanup" Jul 09 23:16:41.362418 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.362350 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dff2859-d928-4979-8d46-03edee62eb9f" containerName="maas-api" Jul 09 23:16:41.362418 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.362359 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0a3c59a-9adc-4e12-bf72-4cd28f8f265a" containerName="maas-api" Jul 09 23:16:41.362418 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.362366 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7620603b-8ed9-447c-96ac-afc67456a0ec" containerName="istio-proxy" Jul 09 23:16:41.362418 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.362373 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7620603b-8ed9-447c-96ac-afc67456a0ec" containerName="istio-proxy" Jul 09 23:16:41.362418 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.362379 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="78a5e362-046e-472c-9385-938de141eacd" containerName="cleanup" Jul 09 23:16:41.365500 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.365461 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" Jul 09 23:16:41.368444 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.368423 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bnh24\"/\"default-dockercfg-f4t74\"" Jul 09 23:16:41.368580 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.368428 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bnh24\"/\"kube-root-ca.crt\"" Jul 09 23:16:41.369471 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.369448 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bnh24\"/\"openshift-service-ca.crt\"" Jul 09 23:16:41.373343 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.373308 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f"] Jul 09 23:16:41.443008 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.442968 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/819cbc6b-c3c0-4b79-adb2-a4670b9e1b04-sys\") pod \"perf-node-gather-daemonset-tx79f\" (UID: \"819cbc6b-c3c0-4b79-adb2-a4670b9e1b04\") " pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" Jul 09 23:16:41.443217 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.443019 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/819cbc6b-c3c0-4b79-adb2-a4670b9e1b04-proc\") pod \"perf-node-gather-daemonset-tx79f\" (UID: \"819cbc6b-c3c0-4b79-adb2-a4670b9e1b04\") " pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" Jul 09 23:16:41.443217 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.443054 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/819cbc6b-c3c0-4b79-adb2-a4670b9e1b04-podres\") pod \"perf-node-gather-daemonset-tx79f\" (UID: \"819cbc6b-c3c0-4b79-adb2-a4670b9e1b04\") " pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" Jul 09 23:16:41.443217 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.443136 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/819cbc6b-c3c0-4b79-adb2-a4670b9e1b04-lib-modules\") pod \"perf-node-gather-daemonset-tx79f\" (UID: \"819cbc6b-c3c0-4b79-adb2-a4670b9e1b04\") " pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" Jul 09 23:16:41.443217 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.443169 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69pds\" (UniqueName: \"kubernetes.io/projected/819cbc6b-c3c0-4b79-adb2-a4670b9e1b04-kube-api-access-69pds\") pod \"perf-node-gather-daemonset-tx79f\" (UID: \"819cbc6b-c3c0-4b79-adb2-a4670b9e1b04\") " pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" Jul 09 23:16:41.544251 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.544209 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/819cbc6b-c3c0-4b79-adb2-a4670b9e1b04-proc\") pod \"perf-node-gather-daemonset-tx79f\" (UID: \"819cbc6b-c3c0-4b79-adb2-a4670b9e1b04\") " pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" Jul 09 23:16:41.544251 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.544249 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/819cbc6b-c3c0-4b79-adb2-a4670b9e1b04-podres\") pod \"perf-node-gather-daemonset-tx79f\" (UID: \"819cbc6b-c3c0-4b79-adb2-a4670b9e1b04\") " pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" Jul 09 23:16:41.544550 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.544286 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/819cbc6b-c3c0-4b79-adb2-a4670b9e1b04-lib-modules\") pod \"perf-node-gather-daemonset-tx79f\" (UID: \"819cbc6b-c3c0-4b79-adb2-a4670b9e1b04\") " pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" Jul 09 23:16:41.544550 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.544364 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/819cbc6b-c3c0-4b79-adb2-a4670b9e1b04-proc\") pod \"perf-node-gather-daemonset-tx79f\" (UID: \"819cbc6b-c3c0-4b79-adb2-a4670b9e1b04\") " pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" Jul 09 23:16:41.544550 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.544396 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/819cbc6b-c3c0-4b79-adb2-a4670b9e1b04-podres\") pod \"perf-node-gather-daemonset-tx79f\" (UID: \"819cbc6b-c3c0-4b79-adb2-a4670b9e1b04\") " pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" Jul 09 23:16:41.544550 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.544394 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/819cbc6b-c3c0-4b79-adb2-a4670b9e1b04-lib-modules\") pod \"perf-node-gather-daemonset-tx79f\" (UID: \"819cbc6b-c3c0-4b79-adb2-a4670b9e1b04\") " pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" Jul 09 23:16:41.544550 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.544359 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69pds\" (UniqueName: \"kubernetes.io/projected/819cbc6b-c3c0-4b79-adb2-a4670b9e1b04-kube-api-access-69pds\") pod \"perf-node-gather-daemonset-tx79f\" (UID: \"819cbc6b-c3c0-4b79-adb2-a4670b9e1b04\") " pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" Jul 09 23:16:41.544550 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.544549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/819cbc6b-c3c0-4b79-adb2-a4670b9e1b04-sys\") pod \"perf-node-gather-daemonset-tx79f\" (UID: \"819cbc6b-c3c0-4b79-adb2-a4670b9e1b04\") " pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" Jul 09 23:16:41.544756 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.544635 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/819cbc6b-c3c0-4b79-adb2-a4670b9e1b04-sys\") pod \"perf-node-gather-daemonset-tx79f\" (UID: \"819cbc6b-c3c0-4b79-adb2-a4670b9e1b04\") " pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" Jul 09 23:16:41.552400 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.552376 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69pds\" (UniqueName: \"kubernetes.io/projected/819cbc6b-c3c0-4b79-adb2-a4670b9e1b04-kube-api-access-69pds\") pod \"perf-node-gather-daemonset-tx79f\" (UID: \"819cbc6b-c3c0-4b79-adb2-a4670b9e1b04\") " pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" Jul 09 23:16:41.678164 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.678073 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" Jul 09 23:16:41.817925 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:41.817898 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f"] Jul 09 23:16:41.819740 ip-10-0-135-80 kubenswrapper[2568]: W0709 23:16:41.819718 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod819cbc6b_c3c0_4b79_adb2_a4670b9e1b04.slice/crio-d87e88c77cd8c3050a5b2ffedc77c6e65fdaa3fe97d1a0444aa7c9c79d90ed03 WatchSource:0}: Error finding container d87e88c77cd8c3050a5b2ffedc77c6e65fdaa3fe97d1a0444aa7c9c79d90ed03: Status 404 returned error can't find the container with id d87e88c77cd8c3050a5b2ffedc77c6e65fdaa3fe97d1a0444aa7c9c79d90ed03 Jul 09 23:16:42.084568 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:42.084525 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" event={"ID":"819cbc6b-c3c0-4b79-adb2-a4670b9e1b04","Type":"ContainerStarted","Data":"8187ecc6aec4904998df956b16898d57e95ca74e1bef577bec68361f1f28a42e"} Jul 09 23:16:42.084763 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:42.084587 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" event={"ID":"819cbc6b-c3c0-4b79-adb2-a4670b9e1b04","Type":"ContainerStarted","Data":"d87e88c77cd8c3050a5b2ffedc77c6e65fdaa3fe97d1a0444aa7c9c79d90ed03"} Jul 09 23:16:42.084763 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:42.084635 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" Jul 09 23:16:42.099958 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:42.099900 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" podStartSLOduration=1.0998836889999999 podStartE2EDuration="1.099883689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:16:42.099194082 +0000 UTC m=+2626.640272878" watchObservedRunningTime="2026-07-09 23:16:42.099883689 +0000 UTC m=+2626.640962483" Jul 09 23:16:43.800870 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:43.800839 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mzf9t_e818597c-2b00-4cd2-9d5a-9af19eca0605/dns/0.log" Jul 09 23:16:43.819414 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:43.819383 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mzf9t_e818597c-2b00-4cd2-9d5a-9af19eca0605/kube-rbac-proxy/0.log" Jul 09 23:16:43.936702 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:43.936675 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sgpbm_85d0d04e-7886-4803-82a9-980ae8b0d714/dns-node-resolver/0.log" Jul 09 23:16:44.514312 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:44.514275 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vsrl6_b8d3fdbc-afb4-486f-a376-b4e70e32bdd8/node-ca/0.log" Jul 09 23:16:45.429558 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:45.429529 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4lz5v8_c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f/istio-proxy/0.log" Jul 09 23:16:45.541599 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:45.541563 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-j48wx_c49e1048-733d-41a3-b184-48173c9132bb/discovery/0.log" Jul 09 23:16:46.422768 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:46.422736 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dzhcj_4ed629cc-bb6f-4797-893b-27ed35d60fb5/serve-healthcheck-canary/0.log" Jul 09 23:16:46.970344 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:46.970315 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p2c2c_3658c581-11c5-4dc1-b603-ea68054b7373/kube-rbac-proxy/0.log" Jul 09 23:16:46.989317 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:46.989285 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p2c2c_3658c581-11c5-4dc1-b603-ea68054b7373/exporter/0.log" Jul 09 23:16:47.009607 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:47.009579 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p2c2c_3658c581-11c5-4dc1-b603-ea68054b7373/extractor/0.log" Jul 09 23:16:48.099971 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:48.099941 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bnh24/perf-node-gather-daemonset-tx79f" Jul 09 23:16:49.574941 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:49.574910 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f9f7f5566-m4mbf_f21b9db1-27c9-4415-83f6-fd1c8be98acd/manager/0.log" Jul 09 23:16:49.849694 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:49.849594 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54dbdc554f-rk8z8_cdae79af-b625-4446-ae94-0b79e35df8fa/manager/0.log" Jul 09 23:16:49.871698 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:49.871664 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54dbdc554f-rk8z8_cdae79af-b625-4446-ae94-0b79e35df8fa/copy-manifests/0.log" Jul 09 23:16:51.069227 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:51.069174 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5f8b5887bc-jq4jc_da80d4dd-0f06-4e5a-be7c-7f7f95cfed92/manager/0.log" Jul 09 23:16:51.092342 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:51.092304 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-4m79d_2767fc5c-f34b-4b29-89fc-fbb43a88df73/openshift-lws-operator/0.log" Jul 09 23:16:51.687159 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:51.687073 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4lz5v8_c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f/istio-proxy/0.log" Jul 09 23:16:51.790061 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:51.790028 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-j48wx_c49e1048-733d-41a3-b184-48173c9132bb/discovery/0.log" Jul 09 23:16:52.993419 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:52.993384 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4lz5v8_c1c71d63-b3ab-40d7-a95e-6de7a9e81a4f/istio-proxy/0.log" Jul 09 23:16:53.102131 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:53.102090 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-j48wx_c49e1048-733d-41a3-b184-48173c9132bb/discovery/0.log" Jul 09 23:16:57.311254 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:57.311227 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8ddvw_ca43f479-c01f-49e4-b1fe-a36149b29f51/kube-multus-additional-cni-plugins/0.log" Jul 09 23:16:57.331585 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:57.331548 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8ddvw_ca43f479-c01f-49e4-b1fe-a36149b29f51/egress-router-binary-copy/0.log" Jul 09 23:16:57.350647 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:57.350621 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8ddvw_ca43f479-c01f-49e4-b1fe-a36149b29f51/cni-plugins/0.log" Jul 09 23:16:57.372787 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:57.372763 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8ddvw_ca43f479-c01f-49e4-b1fe-a36149b29f51/bond-cni-plugin/0.log" Jul 09 23:16:57.393573 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:57.393544 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8ddvw_ca43f479-c01f-49e4-b1fe-a36149b29f51/routeoverride-cni/0.log" Jul 09 23:16:57.413457 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:57.413433 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8ddvw_ca43f479-c01f-49e4-b1fe-a36149b29f51/whereabouts-cni-bincopy/0.log" Jul 09 23:16:57.432086 ip-10-0-135-80 kubenswrapper[2568]: I0709 23:16:57.432058 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8ddvw_ca43f479-c01f-49e4-b1fe-a36149b29f51/whereabouts-cni/0.log"