Jul 16 16:51:02.387370 ip-10-0-136-129 systemd[1]: Starting Kubernetes Kubelet... Jul 16 16:51:02.776138 ip-10-0-136-129 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 16:51:02.776138 ip-10-0-136-129 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 16:51:02.776138 ip-10-0-136-129 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 16:51:02.776138 ip-10-0-136-129 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 16:51:02.776138 ip-10-0-136-129 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 16:51:02.776138 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.710440 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 16:51:02.776138 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.717722 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 16:51:02.776138 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.717973 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 16:51:02.776138 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.717993 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 16:51:02.776138 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.717998 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 16:51:02.776138 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718006 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 16:51:02.776138 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718012 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 16:51:02.776138 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718017 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 16:51:02.776138 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718023 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718027 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718033 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718037 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718042 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718046 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718050 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718054 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718058 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718063 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718067 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718071 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718078 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718082 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718087 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718106 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718111 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718115 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718120 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718124 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 16:51:02.777365 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718128 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718132 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718137 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718141 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718160 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718169 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718174 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718179 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718183 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718187 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718192 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718196 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718200 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718204 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718207 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718211 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718215 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718219 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718223 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718228 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 16:51:02.777981 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718231 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 16:51:02.778610 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718236 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 16:51:02.778610 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718240 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 16:51:02.778610 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718243 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 16:51:02.778610 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718247 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 16:51:02.778610 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718251 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 16:51:02.778610 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718255 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 16:51:02.778610 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718259 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 16:51:02.778610 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718263 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 16:51:02.778610 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718268 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 16:51:02.778610 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718273 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 16:51:02.778610 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718276 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 16:51:02.778610 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718280 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 16:51:02.778610 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718284 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 16:51:02.778610 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718289 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 16:51:02.778610 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718292 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 16:51:02.778610 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718296 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 16:51:02.778610 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718301 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 16:51:02.778610 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718305 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 16:51:02.778610 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718309 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718313 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718317 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718320 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718327 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718331 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718334 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718339 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718343 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718349 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718354 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718359 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718363 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718366 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718370 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718374 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718378 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718382 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718387 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718391 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 16:51:02.779166 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718395 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718399 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718402 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718406 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718410 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718415 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718418 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718422 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718427 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718431 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718434 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718439 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.718443 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719522 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719532 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719537 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719542 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719546 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719550 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719553 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 16:51:02.779706 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719558 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 16:51:02.780241 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719562 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 16:51:02.780241 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719566 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 16:51:02.780241 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719570 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 16:51:02.780241 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719576 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 16:51:02.780241 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719581 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 16:51:02.780241 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719585 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 16:51:02.780241 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719590 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 16:51:02.780241 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719594 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 16:51:02.780241 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719598 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 16:51:02.780241 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719602 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 16:51:02.780241 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719606 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 16:51:02.780241 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719609 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 16:51:02.780241 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719613 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 16:51:02.780241 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719617 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 16 16:51:02.780241 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719621 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 16:51:02.780241 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719626 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 16:51:02.780241 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719630 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 16:51:02.780241 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719634 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 16:51:02.780241 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719637 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 16:51:02.780820 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719643 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 16:51:02.780820 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719648 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 16:51:02.780820 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719653 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 16:51:02.780820 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719657 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 16:51:02.780820 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719662 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 16:51:02.780820 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719669 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 16:51:02.780820 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719673 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 16:51:02.780820 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719677 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 16:51:02.780820 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719681 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 16:51:02.780820 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719685 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 16:51:02.780820 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719689 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 16:51:02.780820 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719694 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 16:51:02.780820 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719698 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 16:51:02.780820 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719701 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 16:51:02.780820 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719705 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 16:51:02.780820 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719709 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 16:51:02.780820 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719713 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 16:51:02.780820 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719717 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 16:51:02.780820 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719721 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 16:51:02.781551 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719725 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 16:51:02.781551 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719728 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 16:51:02.781551 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719733 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 16:51:02.781551 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719737 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 16:51:02.781551 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719741 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 16:51:02.781551 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719744 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 16:51:02.781551 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719748 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 16:51:02.781551 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719751 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 16:51:02.781551 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719755 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 16:51:02.781551 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719759 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 16:51:02.781551 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719762 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 16:51:02.781551 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719766 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 16:51:02.781551 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719770 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 16:51:02.781551 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719774 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 16:51:02.781551 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719778 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 16:51:02.781551 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719782 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 16:51:02.781551 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719786 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 16:51:02.781551 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719790 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 16:51:02.781551 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719794 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719799 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719802 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719806 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719810 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719814 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719818 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719823 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719827 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719833 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719838 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719841 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719845 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719850 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719853 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719857 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719861 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719864 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719868 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719872 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 16:51:02.782439 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719876 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 16:51:02.783571 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719879 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 16:51:02.783571 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719883 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 16:51:02.783571 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719887 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 16:51:02.783571 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719890 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 16:51:02.783571 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719894 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 16:51:02.783571 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719898 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 16:51:02.783571 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719902 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 16:51:02.783571 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719906 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 16:51:02.783571 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719909 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 16:51:02.783571 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719913 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 16:51:02.783571 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719918 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 16:51:02.783571 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719922 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 16:51:02.783571 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719930 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 16:51:02.783571 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.719933 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 16:51:02.783571 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720581 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 16:51:02.783571 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720589 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 16:51:02.783571 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720594 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 16:51:02.783571 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720598 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 16:51:02.783571 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720602 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 16:51:02.784446 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720606 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 16:51:02.784446 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720610 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 16:51:02.784446 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720614 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 16:51:02.784446 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720618 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 16:51:02.784446 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720623 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 16:51:02.784446 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720626 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 16:51:02.784446 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720630 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 16:51:02.784446 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720634 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 16:51:02.784446 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720638 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 16:51:02.784446 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720644 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 16:51:02.784446 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720648 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 16:51:02.784446 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720654 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 16:51:02.784446 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720658 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 16:51:02.784446 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720662 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 16:51:02.784446 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720666 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 16:51:02.784446 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720670 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 16:51:02.784446 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720674 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 16:51:02.784446 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720678 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 16:51:02.784446 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720682 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 16:51:02.785308 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720686 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 16:51:02.785308 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720690 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 16:51:02.785308 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720694 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 16:51:02.785308 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720697 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 16:51:02.785308 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720701 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 16:51:02.785308 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720705 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 16:51:02.785308 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720715 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 16:51:02.785308 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720719 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 16:51:02.785308 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720723 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 16:51:02.785308 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720726 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 16:51:02.785308 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720731 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 16:51:02.785308 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720734 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 16:51:02.785308 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720740 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 16:51:02.785308 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720745 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 16:51:02.785308 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720751 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 16:51:02.785308 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720756 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 16:51:02.785308 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720760 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 16:51:02.785308 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720765 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 16 16:51:02.785308 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720769 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720772 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720777 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720780 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720784 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720787 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720792 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720796 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720799 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720803 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720807 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720810 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720814 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720818 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720822 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720826 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720829 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720833 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720837 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720840 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 16:51:02.786390 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720847 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720857 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720862 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720865 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720869 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720873 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720876 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720880 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720886 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720890 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720893 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720897 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720901 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720905 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720908 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720912 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720916 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720919 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720923 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720926 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 16:51:02.787265 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720930 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720933 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720937 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720941 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720944 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720948 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720952 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720957 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720962 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720965 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720969 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720972 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720977 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720981 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720986 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720990 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.720994 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721082 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721115 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721127 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721135 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 16:51:02.788357 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721142 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721147 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721153 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721159 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721164 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721168 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721174 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721178 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721182 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721187 2575 flags.go:64] FLAG: --cgroup-root="" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721191 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721195 2575 flags.go:64] FLAG: --client-ca-file="" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721199 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721204 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721211 2575 flags.go:64] FLAG: --cluster-domain="" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721215 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721220 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721224 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721229 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721243 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721247 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721252 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721256 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721262 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 16 16:51:02.789247 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721266 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721271 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721276 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721281 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721287 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721292 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721297 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721302 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721307 2575 flags.go:64] FLAG: --enable-server="true" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721311 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721318 2575 flags.go:64] FLAG: --event-burst="100" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721323 2575 flags.go:64] FLAG: --event-qps="50" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721326 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721331 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721335 2575 flags.go:64] FLAG: --eviction-hard="" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721341 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721345 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721349 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721353 2575 flags.go:64] FLAG: --eviction-soft="" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721358 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721362 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721367 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721371 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721376 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721380 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 16:51:02.790333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721384 2575 flags.go:64] FLAG: --feature-gates="" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721389 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721394 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721398 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721402 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721407 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721413 2575 flags.go:64] FLAG: --help="false" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721417 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-136-129.ec2.internal" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721421 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721426 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721430 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721435 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721440 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721444 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721450 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721454 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721459 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721463 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721468 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721472 2575 flags.go:64] FLAG: --kube-reserved="" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721476 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721480 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721485 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721490 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 16:51:02.791433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721495 2575 flags.go:64] FLAG: --lock-file="" Jul 16 16:51:02.791415 ip-10-0-136-129 systemd[1]: Started Kubernetes Kubelet. Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721500 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721504 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721508 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721516 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721520 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721524 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721528 2575 flags.go:64] FLAG: --logging-format="text" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721532 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721537 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721541 2575 flags.go:64] FLAG: --manifest-url="" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721546 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721551 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721555 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721563 2575 flags.go:64] FLAG: --max-pods="110" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721567 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721571 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721575 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721579 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721584 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721588 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721593 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721603 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721608 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721613 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 16:51:02.792722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721618 2575 flags.go:64] FLAG: --pod-cidr="" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721623 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721631 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721635 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721639 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721643 2575 flags.go:64] FLAG: --port="10250" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721648 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721652 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d0336221cd8a7153" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721656 2575 flags.go:64] FLAG: --qos-reserved="" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721661 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721666 2575 flags.go:64] FLAG: --register-node="true" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721670 2575 flags.go:64] FLAG: --register-with-taints="" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721676 2575 flags.go:64] FLAG: --registry-burst="10" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721680 2575 flags.go:64] FLAG: --registry-qps="5" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721684 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721688 2575 flags.go:64] FLAG: --reserved-memory="" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721694 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721698 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721702 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721707 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721711 2575 flags.go:64] FLAG: --runonce="false" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721715 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721721 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721725 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721730 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 16:51:02.915014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721734 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721738 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721743 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721748 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721753 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721757 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721761 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721765 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721770 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721774 2575 flags.go:64] FLAG: --system-cgroups="" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721778 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721786 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721790 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721795 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721801 2575 flags.go:64] FLAG: --tls-min-version="" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721805 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721809 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721813 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721817 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721821 2575 flags.go:64] FLAG: --v="2" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721827 2575 flags.go:64] FLAG: --version="false" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721834 2575 flags.go:64] FLAG: --vmodule="" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721839 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.721845 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722033 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 16:51:02.937485 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722038 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 16:51:02.939254 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722043 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 16:51:02.939254 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722049 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 16:51:02.939254 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722054 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 16:51:02.939254 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722060 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 16:51:02.939254 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722065 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 16:51:02.939254 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722069 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 16:51:02.939254 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722073 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 16:51:02.939254 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722077 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 16:51:02.939254 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722081 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 16:51:02.939254 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722087 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 16:51:02.939254 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722112 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 16:51:02.939254 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722116 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 16:51:02.939254 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722121 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 16:51:02.939254 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722125 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 16:51:02.939254 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722130 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 16:51:02.939254 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722134 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 16:51:02.939254 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722140 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 16:51:02.939254 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722145 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 16:51:02.939254 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722149 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722154 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722158 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722161 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722166 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722170 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722174 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722178 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722182 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722185 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722190 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722194 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722198 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722202 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722206 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722210 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722214 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722219 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722223 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722227 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 16:51:02.939919 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722231 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 16:51:02.940536 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722235 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 16:51:02.940536 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722239 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 16:51:02.940536 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722243 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 16:51:02.940536 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722249 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 16:51:02.940536 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722253 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 16:51:02.940536 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722257 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 16:51:02.940536 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722261 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 16:51:02.940536 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722265 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 16:51:02.940536 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722268 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 16:51:02.940536 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722272 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 16:51:02.940536 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722276 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 16:51:02.940536 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722281 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 16:51:02.940536 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722287 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 16:51:02.940536 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722292 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 16:51:02.940536 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722297 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 16:51:02.940536 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722302 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 16:51:02.940536 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722306 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 16:51:02.940536 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722310 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 16:51:02.940536 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722314 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722317 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722321 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722325 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722328 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722332 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722336 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722340 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722343 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722347 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722354 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722358 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722361 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722365 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722369 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722373 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722377 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722382 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722387 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722391 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 16:51:02.941195 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722395 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 16:51:02.941927 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722398 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 16:51:02.941927 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722402 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 16:51:02.941927 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722406 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 16:51:02.941927 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722411 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 16:51:02.941927 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722415 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 16:51:02.941927 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722418 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 16:51:02.941927 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722422 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 16:51:02.941927 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722426 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 16:51:02.941927 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722430 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 16:51:02.941927 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722434 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 16:51:02.941927 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722438 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 16:51:02.941927 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722442 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 16:51:02.941927 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722446 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 16:51:02.941927 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722450 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 16:51:02.941927 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722454 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 16:51:02.941927 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722457 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 16:51:02.941927 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722461 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 16:51:02.941927 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722465 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 16:51:02.941927 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.722469 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 16:51:02.942620 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.722480 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 16:51:02.942620 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.731450 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 16:51:02.942620 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.731468 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 16:51:02.942620 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.731489 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 16:51:02.942620 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.731498 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 16:51:02.942620 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731556 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 16:51:02.942620 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731562 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 16:51:02.942620 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731566 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 16:51:02.942620 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731569 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 16:51:02.942620 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731573 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 16:51:02.942620 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731577 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 16:51:02.942620 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731581 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 16:51:02.942620 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731584 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 16:51:02.942620 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731587 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731591 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731594 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731596 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731599 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731602 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731605 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731608 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731610 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731613 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731616 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731618 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731621 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731623 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731626 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731628 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731631 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731633 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731636 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731639 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 16:51:02.943418 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731642 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731644 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731647 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731650 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731652 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731655 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731657 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731660 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731662 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731665 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731667 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731671 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731674 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731677 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731679 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731682 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731684 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731687 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731690 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731693 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 16:51:02.944046 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731695 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 16:51:02.944890 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731698 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 16:51:02.944890 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731702 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 16:51:02.944890 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731706 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 16:51:02.944890 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731709 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 16:51:02.944890 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731712 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 16:51:02.944890 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731715 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 16:51:02.944890 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731718 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 16:51:02.944890 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731721 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 16:51:02.944890 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731723 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 16:51:02.944890 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731726 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 16:51:02.944890 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731731 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 16:51:02.944890 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731733 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 16:51:02.944890 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731736 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 16:51:02.944890 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731738 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 16:51:02.944890 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731741 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 16:51:02.944890 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731744 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 16:51:02.944890 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731746 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 16:51:02.944890 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731749 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 16:51:02.944890 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731751 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731754 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731759 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731762 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731765 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731767 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731770 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731772 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731775 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731777 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731780 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731783 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731786 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731789 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731791 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731794 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731796 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731799 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731802 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731804 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 16:51:02.945504 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731807 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 16:51:02.946132 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731809 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 16:51:02.946132 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731812 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 16:51:02.946132 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731815 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 16:51:02.946132 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731819 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 16 16:51:02.946132 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731822 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 16:51:02.946132 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731825 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 16:51:02.946132 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731827 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 16:51:02.946132 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731830 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 16:51:02.946132 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731832 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 16:51:02.946132 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731835 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 16:51:02.946132 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731838 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 16:51:02.946132 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.731843 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 16:51:02.946132 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731972 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 16:51:02.946132 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731978 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 16:51:02.946132 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731983 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 16:51:02.946725 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731986 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 16:51:02.946725 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731989 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 16:51:02.946725 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731992 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 16:51:02.946725 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731995 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 16:51:02.946725 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.731998 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 16:51:02.946725 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732000 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 16:51:02.946725 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732003 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 16:51:02.946725 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732006 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 16:51:02.946725 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732008 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 16:51:02.946725 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732012 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 16:51:02.946725 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732015 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 16:51:02.946725 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732018 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 16:51:02.946725 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732021 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 16:51:02.946725 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732024 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 16:51:02.946725 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732026 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 16:51:02.946725 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732029 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 16:51:02.946725 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732032 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 16:51:02.946725 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732034 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 16:51:02.946725 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732037 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 16:51:02.947324 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732039 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 16:51:02.947324 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732043 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 16:51:02.947324 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732046 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 16:51:02.947324 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732048 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 16:51:02.947324 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732051 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 16:51:02.947324 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732053 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 16:51:02.947324 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732056 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 16:51:02.947324 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732058 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 16:51:02.947324 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732061 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 16:51:02.947324 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732063 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 16:51:02.947324 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732067 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 16:51:02.947324 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732070 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 16:51:02.947324 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732074 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 16:51:02.947324 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732077 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 16:51:02.947324 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732079 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 16:51:02.947324 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732082 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 16:51:02.947324 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732084 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 16:51:02.947324 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732087 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 16:51:02.947324 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732090 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 16:51:02.947849 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732111 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 16:51:02.947849 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732114 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 16:51:02.947849 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732118 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 16:51:02.947849 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732122 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 16:51:02.947849 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732126 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 16:51:02.947849 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732129 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 16:51:02.947849 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732131 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 16:51:02.947849 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732134 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 16:51:02.947849 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732137 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 16:51:02.947849 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732140 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 16:51:02.947849 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732144 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 16:51:02.947849 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732148 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 16:51:02.947849 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732150 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 16:51:02.947849 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732153 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 16:51:02.947849 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732157 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 16:51:02.947849 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732160 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 16:51:02.947849 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732162 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 16:51:02.947849 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732165 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 16:51:02.947849 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732167 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732170 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732173 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732175 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732178 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732180 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732183 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732186 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732189 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732191 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732194 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732196 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732199 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732201 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732204 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732206 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732209 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732211 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732214 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732217 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 16:51:02.948389 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732220 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 16:51:02.948941 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732222 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 16:51:02.948941 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732225 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 16:51:02.948941 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732227 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 16:51:02.948941 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732230 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 16:51:02.948941 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732232 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 16:51:02.948941 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732235 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 16:51:02.948941 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732237 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 16:51:02.948941 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732241 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 16:51:02.948941 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732243 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 16:51:02.948941 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732246 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 16:51:02.948941 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732249 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 16:51:02.948941 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732251 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 16:51:02.948941 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732254 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 16:51:02.948941 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732256 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 16:51:02.948941 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732259 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 16:51:02.948941 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732261 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 16 16:51:02.948941 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732264 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 16:51:02.948941 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732266 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 16:51:02.948941 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:02.732270 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 16:51:03.080296 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.732275 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 16:51:03.080296 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.732387 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 16:51:03.080296 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.735568 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 16:51:03.080296 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.736848 2575 server.go:1019] "Starting client certificate rotation" Jul 16 16:51:03.080296 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.739596 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 16:51:03.080296 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.740207 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 16:51:03.080296 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.740245 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 16:51:03.080296 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.742053 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 16:51:03.080296 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.751479 2575 log.go:25] "Validated CRI v1 runtime API" Jul 16 16:51:03.080296 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.755806 2575 log.go:25] "Validated CRI v1 image API" Jul 16 16:51:03.080296 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.758204 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 16:51:03.080296 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.760361 2575 fs.go:135] Filesystem UUIDs: map[73408388-ab8c-4101-b843-75f3b30f5807:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ca6952ce-4f28-4657-9d43-0dff39195c99:/dev/nvme0n1p3] Jul 16 16:51:03.081366 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.760376 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 16:51:03.081736 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.765165 2575 manager.go:216] Machine: {Timestamp:2026-07-16 16:51:02.763994074 +0000 UTC m=+0.282719299 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102961 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2219106f18d67c5448ca73a3d3f79e SystemUUID:ec221910-6f18-d67c-5448-ca73a3d3f79e BootID:4d3cc414-6abf-4747-8bd1-3feb16217e59 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:82:d7:a3:39:0b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:82:d7:a3:39:0b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:6b:81:45:25:b7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 16:51:03.081736 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.765284 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 16:51:03.081736 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.765385 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 16:51:03.081736 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.767906 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 16:51:03.081736 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.767936 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-129.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 16:51:03.081736 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.776938 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 16:51:03.081736 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.776960 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 16:51:03.081736 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.776970 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.776990 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.778188 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.778772 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.781899 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.782020 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.782042 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.782053 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.782067 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.782076 2575 kubelet.go:398] "Adding apiserver pod source" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.782085 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.782422 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.782435 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.784260 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.786170 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.786198 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.786343 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.786355 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.786361 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.786367 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.786375 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.786381 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.786850 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.786856 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.786864 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.786869 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.786878 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.786885 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.787608 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.787615 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.790665 2575 server.go:1268] "Started kubelet" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.791025 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.792543 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.793326 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.793361 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.794385 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.798089 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.798633 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:02.800377 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-129.ec2.internal\" not found" Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.800582 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-129.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 16:51:03.082201 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:02.800671 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:02.800669 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-129.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.800997 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.801172 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.801185 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:02.801187 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.801306 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.801313 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.801384 2575 factory.go:55] Registering systemd factory Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.801397 2575 factory.go:223] Registration of the systemd container factory successfully Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.801596 2575 factory.go:153] Registering CRI-O factory Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.801604 2575 factory.go:223] Registration of the crio container factory successfully Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.801641 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.801660 2575 factory.go:103] Registering Raw factory Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.801670 2575 manager.go:1196] Started watching for new ooms in manager Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.801964 2575 manager.go:349] Starting recovery of all containers Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.811775 2575 manager.go:354] Recovery completed Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:02.811941 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-129.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:02.811958 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:02.812053 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-129.ec2.internal.18c2d33d568c5684 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-129.ec2.internal,UID:ip-10-0-136-129.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-129.ec2.internal,},FirstTimestamp:2026-07-16 16:51:02.790641284 +0000 UTC m=+0.309366530,LastTimestamp:2026-07-16 16:51:02.790641284 +0000 UTC m=+0.309366530,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-129.ec2.internal,}" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.819518 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.821611 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.821645 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.821659 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.822233 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.822244 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.822252 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.822258 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.822938 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.822967 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.822977 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:51:03.085776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.824497 2575 policy_none.go:49] "None policy: Start" Jul 16 16:51:03.086743 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.824509 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 16:51:03.086743 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.824520 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 16:51:03.086743 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.826460 2575 policy_none.go:47] "Start" Jul 16 16:51:03.086743 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:02.832960 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-129.ec2.internal.18c2d33d5865256f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-129.ec2.internal,UID:ip-10-0-136-129.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-129.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-129.ec2.internal,},FirstTimestamp:2026-07-16 16:51:02.821627247 +0000 UTC m=+0.340352490,LastTimestamp:2026-07-16 16:51:02.821627247 +0000 UTC m=+0.340352490,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-129.ec2.internal,}" Jul 16 16:51:03.086743 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:02.842147 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-129.ec2.internal.18c2d33d5865840c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-129.ec2.internal,UID:ip-10-0-136-129.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-129.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-129.ec2.internal,},FirstTimestamp:2026-07-16 16:51:02.821651468 +0000 UTC m=+0.340376697,LastTimestamp:2026-07-16 16:51:02.821651468 +0000 UTC m=+0.340376697,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-129.ec2.internal,}" Jul 16 16:51:03.086743 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:02.851741 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-129.ec2.internal.18c2d33d5865b96c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-129.ec2.internal,UID:ip-10-0-136-129.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-129.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-129.ec2.internal,},FirstTimestamp:2026-07-16 16:51:02.821665132 +0000 UTC m=+0.340390361,LastTimestamp:2026-07-16 16:51:02.821665132 +0000 UTC m=+0.340390361,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-129.ec2.internal,}" Jul 16 16:51:03.086743 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.856233 2575 manager.go:337] "Starting Device Plugin manager" Jul 16 16:51:03.086743 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:02.856266 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 16:51:03.086743 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.856275 2575 server.go:87] "Starting device plugin registration server" Jul 16 16:51:03.086743 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.856515 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 16:51:03.086743 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.856538 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 16:51:03.086743 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.856614 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 16:51:03.086743 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.856675 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 16:51:03.086743 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.856682 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 16:51:03.086743 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:02.857283 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 16:51:03.086743 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:02.857333 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-129.ec2.internal\" not found" Jul 16 16:51:03.086743 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.861578 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ng5vm" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:02.867033 2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-129.ec2.internal.18c2d33d5865256f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-129.ec2.internal.18c2d33d5865256f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-129.ec2.internal,UID:ip-10-0-136-129.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-129.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-129.ec2.internal,},FirstTimestamp:2026-07-16 16:51:02.821627247 +0000 UTC m=+0.340352490,LastTimestamp:2026-07-16 16:51:02.822951874 +0000 UTC m=+0.341677099,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-129.ec2.internal,}" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.874331 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ng5vm" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:02.876116 2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-129.ec2.internal.18c2d33d5865840c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-129.ec2.internal.18c2d33d5865840c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-129.ec2.internal,UID:ip-10-0-136-129.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-129.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-129.ec2.internal,},FirstTimestamp:2026-07-16 16:51:02.821651468 +0000 UTC m=+0.340376697,LastTimestamp:2026-07-16 16:51:02.822971719 +0000 UTC m=+0.341696944,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-129.ec2.internal,}" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.901756 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.902906 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.902935 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.902953 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:02.902988 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.905291 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.957392 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.958696 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.958725 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.958741 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.958773 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:02.979884 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:02.979907 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-129.ec2.internal\": node \"ip-10-0-136-129.ec2.internal\" not found" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.003069 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-129.ec2.internal"] Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.003145 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.004012 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.004042 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.004060 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.006513 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:51:03.087407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.006641 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.088070 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.006666 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:51:03.088070 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.007236 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:51:03.088070 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.007251 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:51:03.088070 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.007263 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:51:03.088070 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.007271 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:51:03.088070 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.007274 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:51:03.088070 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.007281 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:51:03.088070 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.009593 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.088070 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.009621 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:51:03.088070 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.010508 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:51:03.088070 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.010535 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:51:03.088070 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.010545 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:51:03.088070 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:03.026565 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-129.ec2.internal\" not found" Jul 16 16:51:03.088070 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:03.059080 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-129.ec2.internal\" not found" node="ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.088070 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:03.063780 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-129.ec2.internal\" not found" node="ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.255977 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.102501 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c90abb462952f54ad05f349a2dfb0ab3-config\") pod \"kube-apiserver-proxy-ip-10-0-136-129.ec2.internal\" (UID: \"c90abb462952f54ad05f349a2dfb0ab3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.255977 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.102524 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6563477dbb75145720caa9d8c4e6e99-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal\" (UID: \"b6563477dbb75145720caa9d8c4e6e99\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.255977 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.102551 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6563477dbb75145720caa9d8c4e6e99-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal\" (UID: \"b6563477dbb75145720caa9d8c4e6e99\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.255977 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:03.128533 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-129.ec2.internal\" not found" Jul 16 16:51:03.255977 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.203671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6563477dbb75145720caa9d8c4e6e99-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal\" (UID: \"b6563477dbb75145720caa9d8c4e6e99\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.255977 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.203711 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6563477dbb75145720caa9d8c4e6e99-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal\" (UID: \"b6563477dbb75145720caa9d8c4e6e99\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.255977 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.203740 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c90abb462952f54ad05f349a2dfb0ab3-config\") pod \"kube-apiserver-proxy-ip-10-0-136-129.ec2.internal\" (UID: \"c90abb462952f54ad05f349a2dfb0ab3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.255977 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.203809 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c90abb462952f54ad05f349a2dfb0ab3-config\") pod \"kube-apiserver-proxy-ip-10-0-136-129.ec2.internal\" (UID: \"c90abb462952f54ad05f349a2dfb0ab3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.255977 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.203880 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6563477dbb75145720caa9d8c4e6e99-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal\" (UID: \"b6563477dbb75145720caa9d8c4e6e99\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.255977 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.203921 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6563477dbb75145720caa9d8c4e6e99-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal\" (UID: \"b6563477dbb75145720caa9d8c4e6e99\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.255977 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:03.228685 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-129.ec2.internal\" not found" Jul 16 16:51:03.426062 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:03.329586 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-129.ec2.internal\" not found" Jul 16 16:51:03.426062 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.362205 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.426062 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.367466 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.430719 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:03.430471 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-129.ec2.internal\" not found" Jul 16 16:51:03.591031 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:03.531653 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-129.ec2.internal\" not found" Jul 16 16:51:03.721093 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:03.633145 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-129.ec2.internal\" not found" Jul 16 16:51:03.721093 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.684488 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 16:51:03.721093 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.701802 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.724032 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.723624 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 16:51:03.727963 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.725107 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.807042 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.740886 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 16:51:03.807042 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.740992 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 16:51:03.807042 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:03.741045 2575 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a10b599e408b04beeb75964312fedefb-0b44023226f80b0e.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.136.129:48134->32.198.166.99:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-129.ec2.internal" Jul 16 16:51:03.807042 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.741050 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 16:51:03.807042 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.784277 2575 apiserver.go:52] "Watching apiserver" Jul 16 16:51:03.807042 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.801117 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 16:51:03.807042 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.802448 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 16:51:03.807042 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.803401 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-129.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw","openshift-dns/node-resolver-bgc7n","openshift-multus/multus-ntxgv","openshift-multus/network-metrics-daemon-757wf","openshift-network-operator/iptables-alerter-p4mk9","kube-system/global-pull-secret-syncer-6hg5p","openshift-cluster-node-tuning-operator/tuned-fcbqn","openshift-image-registry/node-ca-qvqn5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal","openshift-multus/multus-additional-cni-plugins-6blcn","openshift-network-diagnostics/network-check-target-2czkl","openshift-ovn-kubernetes/ovnkube-node-fj6nb","kube-system/konnectivity-agent-chkfd"] Jul 16 16:51:03.807042 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.806191 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:03.809053 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.808566 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bgc7n" Jul 16 16:51:03.810859 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.810620 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 16:51:03.810859 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.810707 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jjd89\"" Jul 16 16:51:03.810859 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.810825 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 16:51:03.811209 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.810911 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ntxgv" Jul 16 16:51:03.811209 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.811063 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 16:51:03.811209 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.811166 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 16:51:03.813347 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.811712 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 16:51:03.813347 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.811756 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7c876\"" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.813436 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:03.813498 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-757wf" podUID="2bd6202d-398f-4d16-9c02-69dbfe8c67a7" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.814045 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.814135 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.814145 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.814484 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-p6zgn\"" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.815218 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.815653 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p4mk9" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.816459 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.817871 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.817946 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:03.817955 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hg5p" podUID="96a6eb58-3665-40df-a18f-4379194c172c" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.818250 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.818347 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kzknz\"" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.818444 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.820249 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.822745 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qvqn5" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.823351 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.823476 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.823674 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lhmpg\"" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.825070 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.825589 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.825798 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xmqvz\"" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.826023 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.826145 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.827475 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:03.948330 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:03.827522 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2czkl" podUID="37b5e15a-de57-49d6-9c1a-fa42f89f8788" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.827602 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.827707 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qbkqs\"" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.828010 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.829944 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.833031 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-chkfd" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.833566 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c8b42\"" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.833976 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.834661 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.834725 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.834865 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.834914 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.835031 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.836741 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.836754 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.836944 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5btxt\"" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.865102 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r8md7" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.867894 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.877421 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r8md7" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.878667 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 16:46:03 +0000 UTC" deadline="2027-12-10 09:25:35.061804391 +0000 UTC" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.878743 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12280h34m31.18306602s" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.902813 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906386 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-sys-fs\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-cnibin\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906440 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-multus-conf-dir\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:03.949701 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906463 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97fwl\" (UniqueName: \"kubernetes.io/projected/abcbbffd-bca1-417b-b445-81215aaaf8bb-kube-api-access-97fwl\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:03.950652 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906490 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.950652 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906514 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:03.950652 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906539 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s6c9\" (UniqueName: \"kubernetes.io/projected/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-kube-api-access-4s6c9\") pod \"network-metrics-daemon-757wf\" (UID: \"2bd6202d-398f-4d16-9c02-69dbfe8c67a7\") " pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:03.950652 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906561 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abcbbffd-bca1-417b-b445-81215aaaf8bb-tmp\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:03.950652 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906582 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989-host\") pod \"node-ca-qvqn5\" (UID: \"bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989\") " pod="openshift-image-registry/node-ca-qvqn5" Jul 16 16:51:03.950652 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906602 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f47ef2b6-4f33-4cee-9157-117ef5dffc52-cnibin\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:03.950652 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906625 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ed467d2-8e31-4fc1-8631-3972722764e6-ovnkube-config\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.950652 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906648 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ed467d2-8e31-4fc1-8631-3972722764e6-ovnkube-script-lib\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.950652 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-sysctl-d\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:03.950652 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906713 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-run\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:03.950652 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906739 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11336a91-8f10-40c6-943e-9383f8082593-tmp-dir\") pod \"node-resolver-bgc7n\" (UID: \"11336a91-8f10-40c6-943e-9383f8082593\") " pod="openshift-dns/node-resolver-bgc7n" Jul 16 16:51:03.950652 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906756 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-run-netns\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.950652 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-node-log\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.950652 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906807 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-modprobe-d\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:03.951371 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906828 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-sysctl-conf\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:03.951371 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906850 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/11336a91-8f10-40c6-943e-9383f8082593-hosts-file\") pod \"node-resolver-bgc7n\" (UID: \"11336a91-8f10-40c6-943e-9383f8082593\") " pod="openshift-dns/node-resolver-bgc7n" Jul 16 16:51:03.951371 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906870 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-systemd-units\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.951371 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906892 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-os-release\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:03.951371 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906914 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phpx8\" (UniqueName: \"kubernetes.io/projected/19677c47-199e-4898-8f9d-b38ed966d3b4-kube-api-access-phpx8\") pod \"iptables-alerter-p4mk9\" (UID: \"19677c47-199e-4898-8f9d-b38ed966d3b4\") " pod="openshift-network-operator/iptables-alerter-p4mk9" Jul 16 16:51:03.951371 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906935 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-host\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:03.951371 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906959 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vc4n\" (UniqueName: \"kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n\") pod \"network-check-target-2czkl\" (UID: \"37b5e15a-de57-49d6-9c1a-fa42f89f8788\") " pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:03.951371 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.906986 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-kubelet\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.951371 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907025 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.951371 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907041 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ed467d2-8e31-4fc1-8631-3972722764e6-ovn-node-metrics-cert\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.951371 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907057 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c111c675-66c3-42dc-8f46-74644b5907a3-agent-certs\") pod \"konnectivity-agent-chkfd\" (UID: \"c111c675-66c3-42dc-8f46-74644b5907a3\") " pod="kube-system/konnectivity-agent-chkfd" Jul 16 16:51:03.951371 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907071 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-etc-selinux\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:03.951371 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907120 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-multus-socket-dir-parent\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:03.951371 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907145 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-host-var-lib-kubelet\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:03.951981 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907163 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f47ef2b6-4f33-4cee-9157-117ef5dffc52-cni-binary-copy\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:03.951981 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907181 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f47ef2b6-4f33-4cee-9157-117ef5dffc52-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:03.951981 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-run-openvswitch\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.951981 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907221 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-socket-dir\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:03.951981 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907239 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-multus-cni-dir\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:03.951981 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907255 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/90c59dbd-833b-4fd1-9219-0677d51516ec-cni-binary-copy\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:03.951981 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/90c59dbd-833b-4fd1-9219-0677d51516ec-multus-daemon-config\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:03.951981 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907285 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-tuned\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:03.951981 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907304 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-run-systemd\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.951981 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907330 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-etc-openvswitch\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.951981 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907346 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f47ef2b6-4f33-4cee-9157-117ef5dffc52-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:03.951981 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907362 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-system-cni-dir\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:03.951981 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907377 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-host-run-k8s-cni-cncf-io\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:03.951981 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907391 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-host-run-multus-certs\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:03.952694 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907405 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f47ef2b6-4f33-4cee-9157-117ef5dffc52-os-release\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:03.952694 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907420 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c95bx\" (UniqueName: \"kubernetes.io/projected/f47ef2b6-4f33-4cee-9157-117ef5dffc52-kube-api-access-c95bx\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:03.952694 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907435 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-run-ovn\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.952694 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907494 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7dqr\" (UniqueName: \"kubernetes.io/projected/11336a91-8f10-40c6-943e-9383f8082593-kube-api-access-h7dqr\") pod \"node-resolver-bgc7n\" (UID: \"11336a91-8f10-40c6-943e-9383f8082593\") " pod="openshift-dns/node-resolver-bgc7n" Jul 16 16:51:03.952694 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907539 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-slash\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.952694 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907567 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-cni-bin\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.952694 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907596 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-cni-netd\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.952694 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907618 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-registration-dir\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:03.952694 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19677c47-199e-4898-8f9d-b38ed966d3b4-host-slash\") pod \"iptables-alerter-p4mk9\" (UID: \"19677c47-199e-4898-8f9d-b38ed966d3b4\") " pod="openshift-network-operator/iptables-alerter-p4mk9" Jul 16 16:51:03.952694 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907675 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f47ef2b6-4f33-4cee-9157-117ef5dffc52-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:03.952694 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907705 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-log-socket\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.952694 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907735 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88vx7\" (UniqueName: \"kubernetes.io/projected/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-kube-api-access-88vx7\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:03.952694 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907757 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg4dq\" (UniqueName: \"kubernetes.io/projected/90c59dbd-833b-4fd1-9219-0677d51516ec-kube-api-access-gg4dq\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:03.953270 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907780 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs\") pod \"network-metrics-daemon-757wf\" (UID: \"2bd6202d-398f-4d16-9c02-69dbfe8c67a7\") " pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:03.953270 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907800 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-sysconfig\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:03.953270 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpmqf\" (UniqueName: \"kubernetes.io/projected/bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989-kube-api-access-dpmqf\") pod \"node-ca-qvqn5\" (UID: \"bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989\") " pod="openshift-image-registry/node-ca-qvqn5" Jul 16 16:51:03.953270 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f47ef2b6-4f33-4cee-9157-117ef5dffc52-system-cni-dir\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:03.953270 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907866 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c111c675-66c3-42dc-8f46-74644b5907a3-konnectivity-ca\") pod \"konnectivity-agent-chkfd\" (UID: \"c111c675-66c3-42dc-8f46-74644b5907a3\") " pod="kube-system/konnectivity-agent-chkfd" Jul 16 16:51:03.953270 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907887 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-host-run-netns\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:03.953270 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907909 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96a6eb58-3665-40df-a18f-4379194c172c-kubelet-config\") pod \"global-pull-secret-syncer-6hg5p\" (UID: \"96a6eb58-3665-40df-a18f-4379194c172c\") " pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:03.953270 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907937 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-var-lib-kubelet\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:03.953270 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907970 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19677c47-199e-4898-8f9d-b38ed966d3b4-iptables-alerter-script\") pod \"iptables-alerter-p4mk9\" (UID: \"19677c47-199e-4898-8f9d-b38ed966d3b4\") " pod="openshift-network-operator/iptables-alerter-p4mk9" Jul 16 16:51:03.953270 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.907996 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-sys\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:03.953270 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.908024 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc86r\" (UniqueName: \"kubernetes.io/projected/6ed467d2-8e31-4fc1-8631-3972722764e6-kube-api-access-sc86r\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.953270 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.908050 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-host-var-lib-cni-multus\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:03.953270 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.908073 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-etc-kubernetes\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:03.953270 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.908099 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret\") pod \"global-pull-secret-syncer-6hg5p\" (UID: \"96a6eb58-3665-40df-a18f-4379194c172c\") " pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:03.953973 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.908178 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-var-lib-openvswitch\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:03.953973 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.908209 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-device-dir\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:03.953973 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.908236 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-host-var-lib-cni-bin\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:03.953973 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.908264 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-hostroot\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:03.953973 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.908302 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96a6eb58-3665-40df-a18f-4379194c172c-dbus\") pod \"global-pull-secret-syncer-6hg5p\" (UID: \"96a6eb58-3665-40df-a18f-4379194c172c\") " pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:03.953973 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.908331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-kubernetes\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:03.953973 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.908360 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-systemd\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:03.953973 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.908390 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-lib-modules\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:03.953973 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.908415 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989-serviceca\") pod \"node-ca-qvqn5\" (UID: \"bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989\") " pod="openshift-image-registry/node-ca-qvqn5" Jul 16 16:51:03.953973 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:03.908460 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ed467d2-8e31-4fc1-8631-3972722764e6-env-overrides\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.069493 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.008692 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f47ef2b6-4f33-4cee-9157-117ef5dffc52-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:04.069493 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.008727 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-system-cni-dir\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.069493 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.008756 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-host-run-k8s-cni-cncf-io\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.069493 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.008780 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-host-run-multus-certs\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.069493 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.008805 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f47ef2b6-4f33-4cee-9157-117ef5dffc52-os-release\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:04.069493 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.008815 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-system-cni-dir\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.069493 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.008832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c95bx\" (UniqueName: \"kubernetes.io/projected/f47ef2b6-4f33-4cee-9157-117ef5dffc52-kube-api-access-c95bx\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:04.069493 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.008856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-run-ovn\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.069493 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.008883 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7dqr\" (UniqueName: \"kubernetes.io/projected/11336a91-8f10-40c6-943e-9383f8082593-kube-api-access-h7dqr\") pod \"node-resolver-bgc7n\" (UID: \"11336a91-8f10-40c6-943e-9383f8082593\") " pod="openshift-dns/node-resolver-bgc7n" Jul 16 16:51:04.069493 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.008891 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-host-run-k8s-cni-cncf-io\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.069493 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.008909 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-slash\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.069493 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.008953 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f47ef2b6-4f33-4cee-9157-117ef5dffc52-os-release\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:04.069493 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.008959 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-slash\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.069493 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.008956 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-cni-bin\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.069493 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.008971 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-run-ovn\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.071122 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.008883 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-host-run-multus-certs\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.071122 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009001 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-cni-netd\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.071122 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009028 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-registration-dir\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:04.071122 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009040 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-cni-netd\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.071122 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.008991 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-cni-bin\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.071122 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009053 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19677c47-199e-4898-8f9d-b38ed966d3b4-host-slash\") pod \"iptables-alerter-p4mk9\" (UID: \"19677c47-199e-4898-8f9d-b38ed966d3b4\") " pod="openshift-network-operator/iptables-alerter-p4mk9" Jul 16 16:51:04.071122 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009092 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f47ef2b6-4f33-4cee-9157-117ef5dffc52-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:04.071122 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009101 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-registration-dir\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:04.071122 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-log-socket\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.071122 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009138 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19677c47-199e-4898-8f9d-b38ed966d3b4-host-slash\") pod \"iptables-alerter-p4mk9\" (UID: \"19677c47-199e-4898-8f9d-b38ed966d3b4\") " pod="openshift-network-operator/iptables-alerter-p4mk9" Jul 16 16:51:04.071122 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88vx7\" (UniqueName: \"kubernetes.io/projected/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-kube-api-access-88vx7\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:04.071122 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009188 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg4dq\" (UniqueName: \"kubernetes.io/projected/90c59dbd-833b-4fd1-9219-0677d51516ec-kube-api-access-gg4dq\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.071122 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009215 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs\") pod \"network-metrics-daemon-757wf\" (UID: \"2bd6202d-398f-4d16-9c02-69dbfe8c67a7\") " pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:04.071122 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009242 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-sysconfig\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.071807 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009267 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpmqf\" (UniqueName: \"kubernetes.io/projected/bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989-kube-api-access-dpmqf\") pod \"node-ca-qvqn5\" (UID: \"bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989\") " pod="openshift-image-registry/node-ca-qvqn5" Jul 16 16:51:04.071807 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009291 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f47ef2b6-4f33-4cee-9157-117ef5dffc52-system-cni-dir\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:04.071807 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009314 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-log-socket\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.071807 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009336 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-sysconfig\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.071807 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009293 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f47ef2b6-4f33-4cee-9157-117ef5dffc52-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:04.071807 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009316 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c111c675-66c3-42dc-8f46-74644b5907a3-konnectivity-ca\") pod \"konnectivity-agent-chkfd\" (UID: \"c111c675-66c3-42dc-8f46-74644b5907a3\") " pod="kube-system/konnectivity-agent-chkfd" Jul 16 16:51:04.071807 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009377 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-host-run-netns\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.071807 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009373 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f47ef2b6-4f33-4cee-9157-117ef5dffc52-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:04.071807 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009394 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96a6eb58-3665-40df-a18f-4379194c172c-kubelet-config\") pod \"global-pull-secret-syncer-6hg5p\" (UID: \"96a6eb58-3665-40df-a18f-4379194c172c\") " pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:04.071807 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009412 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-var-lib-kubelet\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.071807 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009373 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f47ef2b6-4f33-4cee-9157-117ef5dffc52-system-cni-dir\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:04.071807 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009430 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19677c47-199e-4898-8f9d-b38ed966d3b4-iptables-alerter-script\") pod \"iptables-alerter-p4mk9\" (UID: \"19677c47-199e-4898-8f9d-b38ed966d3b4\") " pod="openshift-network-operator/iptables-alerter-p4mk9" Jul 16 16:51:04.071807 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009474 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-sys\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.071807 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc86r\" (UniqueName: \"kubernetes.io/projected/6ed467d2-8e31-4fc1-8631-3972722764e6-kube-api-access-sc86r\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.072726 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009500 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-host-run-netns\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.072726 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:04.009508 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:04.072726 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009507 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-host-var-lib-cni-multus\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.072726 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009527 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-host-var-lib-cni-multus\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.072726 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009538 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-etc-kubernetes\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.072726 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009556 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret\") pod \"global-pull-secret-syncer-6hg5p\" (UID: \"96a6eb58-3665-40df-a18f-4379194c172c\") " pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:04.072726 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009560 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96a6eb58-3665-40df-a18f-4379194c172c-kubelet-config\") pod \"global-pull-secret-syncer-6hg5p\" (UID: \"96a6eb58-3665-40df-a18f-4379194c172c\") " pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:04.072726 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009589 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-sys\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.072726 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:04.009592 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs podName:2bd6202d-398f-4d16-9c02-69dbfe8c67a7 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:04.50957033 +0000 UTC m=+2.028295544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs") pod "network-metrics-daemon-757wf" (UID: "2bd6202d-398f-4d16-9c02-69dbfe8c67a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:04.072726 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-var-lib-openvswitch\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.072726 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009611 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-etc-kubernetes\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.072726 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009624 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-device-dir\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:04.072726 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009642 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-host-var-lib-cni-bin\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.072726 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009656 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-hostroot\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.072726 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:04.009668 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:04.073328 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009672 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96a6eb58-3665-40df-a18f-4379194c172c-dbus\") pod \"global-pull-secret-syncer-6hg5p\" (UID: \"96a6eb58-3665-40df-a18f-4379194c172c\") " pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:04.073328 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009688 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-kubernetes\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.073328 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009705 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-systemd\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.073328 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:04.009730 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret podName:96a6eb58-3665-40df-a18f-4379194c172c nodeName:}" failed. No retries permitted until 2026-07-16 16:51:04.509715923 +0000 UTC m=+2.028441142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret") pod "global-pull-secret-syncer-6hg5p" (UID: "96a6eb58-3665-40df-a18f-4379194c172c") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:04.073328 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009750 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-host-var-lib-cni-bin\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.073328 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009748 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-hostroot\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.073328 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009789 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-lib-modules\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.073328 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009818 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-var-lib-kubelet\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.073328 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009822 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989-serviceca\") pod \"node-ca-qvqn5\" (UID: \"bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989\") " pod="openshift-image-registry/node-ca-qvqn5" Jul 16 16:51:04.073328 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009839 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96a6eb58-3665-40df-a18f-4379194c172c-dbus\") pod \"global-pull-secret-syncer-6hg5p\" (UID: \"96a6eb58-3665-40df-a18f-4379194c172c\") " pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:04.073328 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009850 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-kubernetes\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.073328 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009852 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ed467d2-8e31-4fc1-8631-3972722764e6-env-overrides\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.073328 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009873 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-systemd\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.073328 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009881 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-sys-fs\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:04.074139 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009898 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-cnibin\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.074139 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009900 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-var-lib-openvswitch\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.074139 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009792 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-device-dir\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:04.074139 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009914 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-multus-conf-dir\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.074139 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009931 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97fwl\" (UniqueName: \"kubernetes.io/projected/abcbbffd-bca1-417b-b445-81215aaaf8bb-kube-api-access-97fwl\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.074139 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009938 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-sys-fs\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:04.074139 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009937 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-lib-modules\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.074139 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009947 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.074139 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009966 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-multus-conf-dir\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.074139 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009967 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:04.074139 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009992 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s6c9\" (UniqueName: \"kubernetes.io/projected/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-kube-api-access-4s6c9\") pod \"network-metrics-daemon-757wf\" (UID: \"2bd6202d-398f-4d16-9c02-69dbfe8c67a7\") " pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:04.074139 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009994 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-cnibin\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.074139 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010007 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abcbbffd-bca1-417b-b445-81215aaaf8bb-tmp\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.074139 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010023 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989-host\") pod \"node-ca-qvqn5\" (UID: \"bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989\") " pod="openshift-image-registry/node-ca-qvqn5" Jul 16 16:51:04.074139 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010037 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f47ef2b6-4f33-4cee-9157-117ef5dffc52-cnibin\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:04.074703 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010044 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.074703 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.009935 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c111c675-66c3-42dc-8f46-74644b5907a3-konnectivity-ca\") pod \"konnectivity-agent-chkfd\" (UID: \"c111c675-66c3-42dc-8f46-74644b5907a3\") " pod="kube-system/konnectivity-agent-chkfd" Jul 16 16:51:04.074703 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010052 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ed467d2-8e31-4fc1-8631-3972722764e6-ovnkube-config\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.074703 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010091 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ed467d2-8e31-4fc1-8631-3972722764e6-ovnkube-script-lib\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.074703 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010109 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-sysctl-d\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.074703 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010126 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:04.074703 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010128 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-run\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.074703 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010141 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989-host\") pod \"node-ca-qvqn5\" (UID: \"bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989\") " pod="openshift-image-registry/node-ca-qvqn5" Jul 16 16:51:04.074703 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010150 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-run\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.074703 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010155 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11336a91-8f10-40c6-943e-9383f8082593-tmp-dir\") pod \"node-resolver-bgc7n\" (UID: \"11336a91-8f10-40c6-943e-9383f8082593\") " pod="openshift-dns/node-resolver-bgc7n" Jul 16 16:51:04.074703 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010203 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-run-netns\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.074703 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010089 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19677c47-199e-4898-8f9d-b38ed966d3b4-iptables-alerter-script\") pod \"iptables-alerter-p4mk9\" (UID: \"19677c47-199e-4898-8f9d-b38ed966d3b4\") " pod="openshift-network-operator/iptables-alerter-p4mk9" Jul 16 16:51:04.074703 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010225 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-node-log\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.074703 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-modprobe-d\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.074703 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010259 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-sysctl-conf\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.075445 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010276 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/11336a91-8f10-40c6-943e-9383f8082593-hosts-file\") pod \"node-resolver-bgc7n\" (UID: \"11336a91-8f10-40c6-943e-9383f8082593\") " pod="openshift-dns/node-resolver-bgc7n" Jul 16 16:51:04.075445 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010291 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-systemd-units\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.075445 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010313 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-os-release\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.075445 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010330 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phpx8\" (UniqueName: \"kubernetes.io/projected/19677c47-199e-4898-8f9d-b38ed966d3b4-kube-api-access-phpx8\") pod \"iptables-alerter-p4mk9\" (UID: \"19677c47-199e-4898-8f9d-b38ed966d3b4\") " pod="openshift-network-operator/iptables-alerter-p4mk9" Jul 16 16:51:04.075445 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010365 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-host\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.075445 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010395 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vc4n\" (UniqueName: \"kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n\") pod \"network-check-target-2czkl\" (UID: \"37b5e15a-de57-49d6-9c1a-fa42f89f8788\") " pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:04.075445 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010426 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-kubelet\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.075445 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.075445 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010459 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ed467d2-8e31-4fc1-8631-3972722764e6-ovn-node-metrics-cert\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.075445 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010479 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c111c675-66c3-42dc-8f46-74644b5907a3-agent-certs\") pod \"konnectivity-agent-chkfd\" (UID: \"c111c675-66c3-42dc-8f46-74644b5907a3\") " pod="kube-system/konnectivity-agent-chkfd" Jul 16 16:51:04.075445 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010458 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 16:51:04.075445 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010205 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f47ef2b6-4f33-4cee-9157-117ef5dffc52-cnibin\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:04.075445 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010495 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-etc-selinux\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:04.075445 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010621 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-multus-socket-dir-parent\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.075445 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010646 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-host-var-lib-kubelet\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.076004 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010660 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-sysctl-d\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.076004 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010678 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f47ef2b6-4f33-4cee-9157-117ef5dffc52-cni-binary-copy\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:04.076004 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010706 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f47ef2b6-4f33-4cee-9157-117ef5dffc52-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:04.076004 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010710 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-node-log\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.076004 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010732 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-run-openvswitch\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.076004 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010750 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-run-netns\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.076004 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-socket-dir\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:04.076004 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-multus-cni-dir\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.076004 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010805 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-systemd-units\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.076004 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/90c59dbd-833b-4fd1-9219-0677d51516ec-cni-binary-copy\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.076004 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010847 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/90c59dbd-833b-4fd1-9219-0677d51516ec-multus-daemon-config\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.076004 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010860 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-host\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.076004 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010883 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-tuned\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.076004 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010908 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-run-systemd\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.076004 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010935 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-etc-openvswitch\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.076720 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010944 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-modprobe-d\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.076720 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.011019 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.076720 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.011048 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-os-release\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.076720 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.011146 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ed467d2-8e31-4fc1-8631-3972722764e6-ovnkube-config\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.076720 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.011147 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-multus-socket-dir-parent\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.076720 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.010507 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/11336a91-8f10-40c6-943e-9383f8082593-hosts-file\") pod \"node-resolver-bgc7n\" (UID: \"11336a91-8f10-40c6-943e-9383f8082593\") " pod="openshift-dns/node-resolver-bgc7n" Jul 16 16:51:04.076720 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.011411 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-host-var-lib-kubelet\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.076720 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.011429 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-run-openvswitch\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.076720 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.011515 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-etc-selinux\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:04.076720 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.011516 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-socket-dir\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:04.076720 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.011576 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f47ef2b6-4f33-4cee-9157-117ef5dffc52-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:04.076720 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.011617 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-sysctl-conf\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.076720 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.011657 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-run-systemd\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.076720 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.011666 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-etc-openvswitch\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.076720 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.011708 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90c59dbd-833b-4fd1-9219-0677d51516ec-multus-cni-dir\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.077269 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.011974 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ed467d2-8e31-4fc1-8631-3972722764e6-host-kubelet\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.077269 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.012049 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/90c59dbd-833b-4fd1-9219-0677d51516ec-cni-binary-copy\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.077269 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.012151 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f47ef2b6-4f33-4cee-9157-117ef5dffc52-cni-binary-copy\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:04.077269 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.012159 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/90c59dbd-833b-4fd1-9219-0677d51516ec-multus-daemon-config\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.077269 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.014188 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/abcbbffd-bca1-417b-b445-81215aaaf8bb-etc-tuned\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.077269 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.014206 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abcbbffd-bca1-417b-b445-81215aaaf8bb-tmp\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.077269 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.014414 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c111c675-66c3-42dc-8f46-74644b5907a3-agent-certs\") pod \"konnectivity-agent-chkfd\" (UID: \"c111c675-66c3-42dc-8f46-74644b5907a3\") " pod="kube-system/konnectivity-agent-chkfd" Jul 16 16:51:04.077269 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.014974 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ed467d2-8e31-4fc1-8631-3972722764e6-ovn-node-metrics-cert\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.077269 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:04.018079 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:51:04.077269 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:04.018114 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:51:04.077269 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:04.018129 2575 projected.go:196] Error preparing data for projected volume kube-api-access-5vc4n for pod openshift-network-diagnostics/network-check-target-2czkl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:04.077269 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:04.018187 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n podName:37b5e15a-de57-49d6-9c1a-fa42f89f8788 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:04.518169177 +0000 UTC m=+2.036894414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5vc4n" (UniqueName: "kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n") pod "network-check-target-2czkl" (UID: "37b5e15a-de57-49d6-9c1a-fa42f89f8788") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:04.077269 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.019970 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpmqf\" (UniqueName: \"kubernetes.io/projected/bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989-kube-api-access-dpmqf\") pod \"node-ca-qvqn5\" (UID: \"bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989\") " pod="openshift-image-registry/node-ca-qvqn5" Jul 16 16:51:04.077269 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.020221 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c95bx\" (UniqueName: \"kubernetes.io/projected/f47ef2b6-4f33-4cee-9157-117ef5dffc52-kube-api-access-c95bx\") pod \"multus-additional-cni-plugins-6blcn\" (UID: \"f47ef2b6-4f33-4cee-9157-117ef5dffc52\") " pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:04.153819 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.020495 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc86r\" (UniqueName: \"kubernetes.io/projected/6ed467d2-8e31-4fc1-8631-3972722764e6-kube-api-access-sc86r\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.153819 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.020643 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88vx7\" (UniqueName: \"kubernetes.io/projected/88bf4ea2-a95a-4fe5-ad54-39972ed01f6f-kube-api-access-88vx7\") pod \"aws-ebs-csi-driver-node-lzkpw\" (UID: \"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:04.153819 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.021354 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg4dq\" (UniqueName: \"kubernetes.io/projected/90c59dbd-833b-4fd1-9219-0677d51516ec-kube-api-access-gg4dq\") pod \"multus-ntxgv\" (UID: \"90c59dbd-833b-4fd1-9219-0677d51516ec\") " pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.153819 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.021659 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phpx8\" (UniqueName: \"kubernetes.io/projected/19677c47-199e-4898-8f9d-b38ed966d3b4-kube-api-access-phpx8\") pod \"iptables-alerter-p4mk9\" (UID: \"19677c47-199e-4898-8f9d-b38ed966d3b4\") " pod="openshift-network-operator/iptables-alerter-p4mk9" Jul 16 16:51:04.153819 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.022093 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97fwl\" (UniqueName: \"kubernetes.io/projected/abcbbffd-bca1-417b-b445-81215aaaf8bb-kube-api-access-97fwl\") pod \"tuned-fcbqn\" (UID: \"abcbbffd-bca1-417b-b445-81215aaaf8bb\") " pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.153819 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.022111 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s6c9\" (UniqueName: \"kubernetes.io/projected/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-kube-api-access-4s6c9\") pod \"network-metrics-daemon-757wf\" (UID: \"2bd6202d-398f-4d16-9c02-69dbfe8c67a7\") " pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:04.153819 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.024948 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ed467d2-8e31-4fc1-8631-3972722764e6-env-overrides\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.153819 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.024983 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989-serviceca\") pod \"node-ca-qvqn5\" (UID: \"bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989\") " pod="openshift-image-registry/node-ca-qvqn5" Jul 16 16:51:04.153819 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.025184 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ed467d2-8e31-4fc1-8631-3972722764e6-ovnkube-script-lib\") pod \"ovnkube-node-fj6nb\" (UID: \"6ed467d2-8e31-4fc1-8631-3972722764e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.153819 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.025157 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11336a91-8f10-40c6-943e-9383f8082593-tmp-dir\") pod \"node-resolver-bgc7n\" (UID: \"11336a91-8f10-40c6-943e-9383f8082593\") " pod="openshift-dns/node-resolver-bgc7n" Jul 16 16:51:04.153819 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.026995 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7dqr\" (UniqueName: \"kubernetes.io/projected/11336a91-8f10-40c6-943e-9383f8082593-kube-api-access-h7dqr\") pod \"node-resolver-bgc7n\" (UID: \"11336a91-8f10-40c6-943e-9383f8082593\") " pod="openshift-dns/node-resolver-bgc7n" Jul 16 16:51:04.153819 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.040066 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:04.153819 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.045072 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-chkfd" Jul 16 16:51:04.153819 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:04.097656 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6563477dbb75145720caa9d8c4e6e99.slice/crio-affe05d4139a3de3476174b56c39d10a48577f9026ed41475daf770be8b323ef WatchSource:0}: Error finding container affe05d4139a3de3476174b56c39d10a48577f9026ed41475daf770be8b323ef: Status 404 returned error can't find the container with id affe05d4139a3de3476174b56c39d10a48577f9026ed41475daf770be8b323ef Jul 16 16:51:04.153819 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.107937 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 16:51:04.153819 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.152983 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" Jul 16 16:51:04.323272 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.158367 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bgc7n" Jul 16 16:51:04.323272 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:04.166751 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88bf4ea2_a95a_4fe5_ad54_39972ed01f6f.slice/crio-814fa1096e00ea62fed414f9da886c07fe73593f51541d9a5d75a6fa20200ef2 WatchSource:0}: Error finding container 814fa1096e00ea62fed414f9da886c07fe73593f51541d9a5d75a6fa20200ef2: Status 404 returned error can't find the container with id 814fa1096e00ea62fed414f9da886c07fe73593f51541d9a5d75a6fa20200ef2 Jul 16 16:51:04.323272 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:04.173083 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11336a91_8f10_40c6_943e_9383f8082593.slice/crio-0e4e85ba3d1a590a606b75d2e3a69e6c330af550de541602f7731d82b5039e9d WatchSource:0}: Error finding container 0e4e85ba3d1a590a606b75d2e3a69e6c330af550de541602f7731d82b5039e9d: Status 404 returned error can't find the container with id 0e4e85ba3d1a590a606b75d2e3a69e6c330af550de541602f7731d82b5039e9d Jul 16 16:51:04.323272 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.183723 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ntxgv" Jul 16 16:51:04.323272 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:04.192514 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c59dbd_833b_4fd1_9219_0677d51516ec.slice/crio-ba45a386f585a42666cc85f252c323fdd41d3980b74803d84acf0908d1fe6fef WatchSource:0}: Error finding container ba45a386f585a42666cc85f252c323fdd41d3980b74803d84acf0908d1fe6fef: Status 404 returned error can't find the container with id ba45a386f585a42666cc85f252c323fdd41d3980b74803d84acf0908d1fe6fef Jul 16 16:51:04.323272 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.232504 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p4mk9" Jul 16 16:51:04.323272 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:04.241320 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19677c47_199e_4898_8f9d_b38ed966d3b4.slice/crio-692abd6b2c52d72b03c5ec218f911dc881f34d5a218e30957f5abeba960f4e97 WatchSource:0}: Error finding container 692abd6b2c52d72b03c5ec218f911dc881f34d5a218e30957f5abeba960f4e97: Status 404 returned error can't find the container with id 692abd6b2c52d72b03c5ec218f911dc881f34d5a218e30957f5abeba960f4e97 Jul 16 16:51:04.323272 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.255731 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" Jul 16 16:51:04.323272 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:04.264900 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabcbbffd_bca1_417b_b445_81215aaaf8bb.slice/crio-e601b689037363f0e450b2bf76b9c9167af881cf8fc7063f188bfe371ee9f5fb WatchSource:0}: Error finding container e601b689037363f0e450b2bf76b9c9167af881cf8fc7063f188bfe371ee9f5fb: Status 404 returned error can't find the container with id e601b689037363f0e450b2bf76b9c9167af881cf8fc7063f188bfe371ee9f5fb Jul 16 16:51:04.323272 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.280193 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qvqn5" Jul 16 16:51:04.323272 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:04.294871 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba4b37b_cf6a_4dde_afdb_eb2b7f9f9989.slice/crio-9c96ea41d429da991e911e3e56a0d2661a993746e7e7a4095822091c0117a820 WatchSource:0}: Error finding container 9c96ea41d429da991e911e3e56a0d2661a993746e7e7a4095822091c0117a820: Status 404 returned error can't find the container with id 9c96ea41d429da991e911e3e56a0d2661a993746e7e7a4095822091c0117a820 Jul 16 16:51:04.323272 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.305491 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6blcn" Jul 16 16:51:04.323272 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:04.317103 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47ef2b6_4f33_4cee_9157_117ef5dffc52.slice/crio-32af1656b7e567e42cd6ee2b7c7551d58e0a06e0e75ff25ad29dd54c6f572ebc WatchSource:0}: Error finding container 32af1656b7e567e42cd6ee2b7c7551d58e0a06e0e75ff25ad29dd54c6f572ebc: Status 404 returned error can't find the container with id 32af1656b7e567e42cd6ee2b7c7551d58e0a06e0e75ff25ad29dd54c6f572ebc Jul 16 16:51:04.473366 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.350668 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 16:51:04.515721 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.515692 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs\") pod \"network-metrics-daemon-757wf\" (UID: \"2bd6202d-398f-4d16-9c02-69dbfe8c67a7\") " pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:04.515862 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.515732 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret\") pod \"global-pull-secret-syncer-6hg5p\" (UID: \"96a6eb58-3665-40df-a18f-4379194c172c\") " pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:04.515862 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:04.515843 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:04.515862 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:04.515850 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:04.515965 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:04.515899 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret podName:96a6eb58-3665-40df-a18f-4379194c172c nodeName:}" failed. No retries permitted until 2026-07-16 16:51:05.515885143 +0000 UTC m=+3.034610356 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret") pod "global-pull-secret-syncer-6hg5p" (UID: "96a6eb58-3665-40df-a18f-4379194c172c") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:04.515965 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:04.515913 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs podName:2bd6202d-398f-4d16-9c02-69dbfe8c67a7 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:05.515906304 +0000 UTC m=+3.034631516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs") pod "network-metrics-daemon-757wf" (UID: "2bd6202d-398f-4d16-9c02-69dbfe8c67a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:04.537416 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:04.537394 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed467d2_8e31_4fc1_8631_3972722764e6.slice/crio-93e988904e12115c2d04a0b6f5bcf65ff8ead595f1de73ab17c316ccfa9ceca2 WatchSource:0}: Error finding container 93e988904e12115c2d04a0b6f5bcf65ff8ead595f1de73ab17c316ccfa9ceca2: Status 404 returned error can't find the container with id 93e988904e12115c2d04a0b6f5bcf65ff8ead595f1de73ab17c316ccfa9ceca2 Jul 16 16:51:04.575357 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:04.575300 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc111c675_66c3_42dc_8f46_74644b5907a3.slice/crio-8e29d8a74321bfba2e016a2e0efcb20ff718a724e01f611c790298b84ba06427 WatchSource:0}: Error finding container 8e29d8a74321bfba2e016a2e0efcb20ff718a724e01f611c790298b84ba06427: Status 404 returned error can't find the container with id 8e29d8a74321bfba2e016a2e0efcb20ff718a724e01f611c790298b84ba06427 Jul 16 16:51:04.616657 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.616630 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vc4n\" (UniqueName: \"kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n\") pod \"network-check-target-2czkl\" (UID: \"37b5e15a-de57-49d6-9c1a-fa42f89f8788\") " pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:04.616765 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:04.616752 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:51:04.616800 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:04.616767 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:51:04.616800 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:04.616777 2575 projected.go:196] Error preparing data for projected volume kube-api-access-5vc4n for pod openshift-network-diagnostics/network-check-target-2czkl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:04.616875 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:04.616823 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n podName:37b5e15a-de57-49d6-9c1a-fa42f89f8788 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:05.616809164 +0000 UTC m=+3.135534377 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vc4n" (UniqueName: "kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n") pod "network-check-target-2czkl" (UID: "37b5e15a-de57-49d6-9c1a-fa42f89f8788") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:04.881019 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.880889 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 16:46:04 +0000 UTC" deadline="2027-12-16 03:03:40.643650471 +0000 UTC" Jul 16 16:51:04.881019 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.880973 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12418h12m35.762682067s" Jul 16 16:51:04.914121 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.913988 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" event={"ID":"6ed467d2-8e31-4fc1-8631-3972722764e6","Type":"ContainerStarted","Data":"93e988904e12115c2d04a0b6f5bcf65ff8ead595f1de73ab17c316ccfa9ceca2"} Jul 16 16:51:04.917598 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.917529 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" event={"ID":"abcbbffd-bca1-417b-b445-81215aaaf8bb","Type":"ContainerStarted","Data":"e601b689037363f0e450b2bf76b9c9167af881cf8fc7063f188bfe371ee9f5fb"} Jul 16 16:51:04.920289 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.920228 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p4mk9" event={"ID":"19677c47-199e-4898-8f9d-b38ed966d3b4","Type":"ContainerStarted","Data":"692abd6b2c52d72b03c5ec218f911dc881f34d5a218e30957f5abeba960f4e97"} Jul 16 16:51:04.927818 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.927790 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" event={"ID":"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f","Type":"ContainerStarted","Data":"814fa1096e00ea62fed414f9da886c07fe73593f51541d9a5d75a6fa20200ef2"} Jul 16 16:51:04.930465 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.930384 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal" event={"ID":"b6563477dbb75145720caa9d8c4e6e99","Type":"ContainerStarted","Data":"affe05d4139a3de3476174b56c39d10a48577f9026ed41475daf770be8b323ef"} Jul 16 16:51:04.933653 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.933508 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6blcn" event={"ID":"f47ef2b6-4f33-4cee-9157-117ef5dffc52","Type":"ContainerStarted","Data":"32af1656b7e567e42cd6ee2b7c7551d58e0a06e0e75ff25ad29dd54c6f572ebc"} Jul 16 16:51:04.937268 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.937223 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qvqn5" event={"ID":"bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989","Type":"ContainerStarted","Data":"9c96ea41d429da991e911e3e56a0d2661a993746e7e7a4095822091c0117a820"} Jul 16 16:51:04.939283 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.939255 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ntxgv" event={"ID":"90c59dbd-833b-4fd1-9219-0677d51516ec","Type":"ContainerStarted","Data":"ba45a386f585a42666cc85f252c323fdd41d3980b74803d84acf0908d1fe6fef"} Jul 16 16:51:04.941352 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.941274 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bgc7n" event={"ID":"11336a91-8f10-40c6-943e-9383f8082593","Type":"ContainerStarted","Data":"0e4e85ba3d1a590a606b75d2e3a69e6c330af550de541602f7731d82b5039e9d"} Jul 16 16:51:04.946562 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.946491 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-chkfd" event={"ID":"c111c675-66c3-42dc-8f46-74644b5907a3","Type":"ContainerStarted","Data":"8e29d8a74321bfba2e016a2e0efcb20ff718a724e01f611c790298b84ba06427"} Jul 16 16:51:04.973454 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:04.973430 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 16:51:05.524856 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:05.524813 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs\") pod \"network-metrics-daemon-757wf\" (UID: \"2bd6202d-398f-4d16-9c02-69dbfe8c67a7\") " pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:05.525054 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:05.524874 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret\") pod \"global-pull-secret-syncer-6hg5p\" (UID: \"96a6eb58-3665-40df-a18f-4379194c172c\") " pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:05.525151 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:05.525049 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:05.525151 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:05.525115 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret podName:96a6eb58-3665-40df-a18f-4379194c172c nodeName:}" failed. No retries permitted until 2026-07-16 16:51:07.525096208 +0000 UTC m=+5.043821437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret") pod "global-pull-secret-syncer-6hg5p" (UID: "96a6eb58-3665-40df-a18f-4379194c172c") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:05.525595 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:05.525573 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:05.525595 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:05.525637 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs podName:2bd6202d-398f-4d16-9c02-69dbfe8c67a7 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:07.525617484 +0000 UTC m=+5.044342698 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs") pod "network-metrics-daemon-757wf" (UID: "2bd6202d-398f-4d16-9c02-69dbfe8c67a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:05.625562 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:05.625525 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vc4n\" (UniqueName: \"kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n\") pod \"network-check-target-2czkl\" (UID: \"37b5e15a-de57-49d6-9c1a-fa42f89f8788\") " pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:05.625723 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:05.625698 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:51:05.625723 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:05.625723 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:51:05.625846 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:05.625736 2575 projected.go:196] Error preparing data for projected volume kube-api-access-5vc4n for pod openshift-network-diagnostics/network-check-target-2czkl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:05.625846 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:05.625796 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n podName:37b5e15a-de57-49d6-9c1a-fa42f89f8788 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:07.625776127 +0000 UTC m=+5.144501355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vc4n" (UniqueName: "kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n") pod "network-check-target-2czkl" (UID: "37b5e15a-de57-49d6-9c1a-fa42f89f8788") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:05.881862 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:05.881736 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 16:46:04 +0000 UTC" deadline="2028-04-04 11:46:36.20117134 +0000 UTC" Jul 16 16:51:05.881862 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:05.881783 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15066h55m30.319392427s" Jul 16 16:51:05.903891 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:05.903841 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:05.903891 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:05.903861 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:05.903891 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:05.903861 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:05.904318 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:05.904013 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-757wf" podUID="2bd6202d-398f-4d16-9c02-69dbfe8c67a7" Jul 16 16:51:05.904318 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:05.904108 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hg5p" podUID="96a6eb58-3665-40df-a18f-4379194c172c" Jul 16 16:51:05.904318 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:05.904246 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2czkl" podUID="37b5e15a-de57-49d6-9c1a-fa42f89f8788" Jul 16 16:51:06.495781 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:06.495753 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 16:51:07.543764 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:07.543717 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret\") pod \"global-pull-secret-syncer-6hg5p\" (UID: \"96a6eb58-3665-40df-a18f-4379194c172c\") " pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:07.544239 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:07.543816 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs\") pod \"network-metrics-daemon-757wf\" (UID: \"2bd6202d-398f-4d16-9c02-69dbfe8c67a7\") " pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:07.544239 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:07.543886 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:07.544239 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:07.543929 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:07.544239 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:07.543959 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret podName:96a6eb58-3665-40df-a18f-4379194c172c nodeName:}" failed. No retries permitted until 2026-07-16 16:51:11.543939081 +0000 UTC m=+9.062664310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret") pod "global-pull-secret-syncer-6hg5p" (UID: "96a6eb58-3665-40df-a18f-4379194c172c") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:07.544239 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:07.543982 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs podName:2bd6202d-398f-4d16-9c02-69dbfe8c67a7 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:11.543965114 +0000 UTC m=+9.062690326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs") pod "network-metrics-daemon-757wf" (UID: "2bd6202d-398f-4d16-9c02-69dbfe8c67a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:07.644481 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:07.644441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vc4n\" (UniqueName: \"kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n\") pod \"network-check-target-2czkl\" (UID: \"37b5e15a-de57-49d6-9c1a-fa42f89f8788\") " pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:07.644668 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:07.644643 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:51:07.644730 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:07.644672 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:51:07.644730 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:07.644686 2575 projected.go:196] Error preparing data for projected volume kube-api-access-5vc4n for pod openshift-network-diagnostics/network-check-target-2czkl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:07.644814 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:07.644756 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n podName:37b5e15a-de57-49d6-9c1a-fa42f89f8788 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:11.644736559 +0000 UTC m=+9.163461776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vc4n" (UniqueName: "kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n") pod "network-check-target-2czkl" (UID: "37b5e15a-de57-49d6-9c1a-fa42f89f8788") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:07.904157 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:07.904123 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:07.904337 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:07.904253 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:07.904337 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:07.904262 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hg5p" podUID="96a6eb58-3665-40df-a18f-4379194c172c" Jul 16 16:51:07.904337 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:07.904280 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:07.904511 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:07.904370 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2czkl" podUID="37b5e15a-de57-49d6-9c1a-fa42f89f8788" Jul 16 16:51:07.904511 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:07.904417 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-757wf" podUID="2bd6202d-398f-4d16-9c02-69dbfe8c67a7" Jul 16 16:51:09.903578 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:09.903529 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:09.904040 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:09.903662 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:09.904040 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:09.903666 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2czkl" podUID="37b5e15a-de57-49d6-9c1a-fa42f89f8788" Jul 16 16:51:09.904040 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:09.903746 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-757wf" podUID="2bd6202d-398f-4d16-9c02-69dbfe8c67a7" Jul 16 16:51:09.904040 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:09.903789 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:09.904040 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:09.903883 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hg5p" podUID="96a6eb58-3665-40df-a18f-4379194c172c" Jul 16 16:51:11.574216 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:11.574174 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs\") pod \"network-metrics-daemon-757wf\" (UID: \"2bd6202d-398f-4d16-9c02-69dbfe8c67a7\") " pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:11.574684 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:11.574233 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret\") pod \"global-pull-secret-syncer-6hg5p\" (UID: \"96a6eb58-3665-40df-a18f-4379194c172c\") " pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:11.574684 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:11.574380 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:11.574684 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:11.574442 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret podName:96a6eb58-3665-40df-a18f-4379194c172c nodeName:}" failed. No retries permitted until 2026-07-16 16:51:19.574424505 +0000 UTC m=+17.093149722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret") pod "global-pull-secret-syncer-6hg5p" (UID: "96a6eb58-3665-40df-a18f-4379194c172c") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:11.574853 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:11.574823 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:11.574899 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:11.574874 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs podName:2bd6202d-398f-4d16-9c02-69dbfe8c67a7 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:19.574857043 +0000 UTC m=+17.093582260 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs") pod "network-metrics-daemon-757wf" (UID: "2bd6202d-398f-4d16-9c02-69dbfe8c67a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:11.675031 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:11.674971 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vc4n\" (UniqueName: \"kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n\") pod \"network-check-target-2czkl\" (UID: \"37b5e15a-de57-49d6-9c1a-fa42f89f8788\") " pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:11.675223 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:11.675153 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:51:11.675223 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:11.675185 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:51:11.675223 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:11.675200 2575 projected.go:196] Error preparing data for projected volume kube-api-access-5vc4n for pod openshift-network-diagnostics/network-check-target-2czkl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:11.675335 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:11.675261 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n podName:37b5e15a-de57-49d6-9c1a-fa42f89f8788 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:19.675239768 +0000 UTC m=+17.193965003 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vc4n" (UniqueName: "kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n") pod "network-check-target-2czkl" (UID: "37b5e15a-de57-49d6-9c1a-fa42f89f8788") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:11.903823 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:11.903787 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:11.903995 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:11.903928 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-757wf" podUID="2bd6202d-398f-4d16-9c02-69dbfe8c67a7" Jul 16 16:51:11.904573 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:11.904321 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:11.904573 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:11.904422 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hg5p" podUID="96a6eb58-3665-40df-a18f-4379194c172c" Jul 16 16:51:11.904573 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:11.904470 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:11.904573 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:11.904531 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2czkl" podUID="37b5e15a-de57-49d6-9c1a-fa42f89f8788" Jul 16 16:51:13.904129 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:13.904077 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:13.904548 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:13.904078 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:13.904548 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:13.904230 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2czkl" podUID="37b5e15a-de57-49d6-9c1a-fa42f89f8788" Jul 16 16:51:13.904548 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:13.904078 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:13.904548 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:13.904327 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-757wf" podUID="2bd6202d-398f-4d16-9c02-69dbfe8c67a7" Jul 16 16:51:13.904548 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:13.904391 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hg5p" podUID="96a6eb58-3665-40df-a18f-4379194c172c" Jul 16 16:51:15.904069 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:15.904036 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:15.904498 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:15.904035 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:15.904498 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:15.904037 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:15.904498 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:15.904178 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-757wf" podUID="2bd6202d-398f-4d16-9c02-69dbfe8c67a7" Jul 16 16:51:15.904498 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:15.904276 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2czkl" podUID="37b5e15a-de57-49d6-9c1a-fa42f89f8788" Jul 16 16:51:15.904498 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:15.904356 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hg5p" podUID="96a6eb58-3665-40df-a18f-4379194c172c" Jul 16 16:51:17.903911 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:17.903870 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:17.903911 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:17.903891 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:17.904450 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:17.903891 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:17.904450 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:17.904009 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-757wf" podUID="2bd6202d-398f-4d16-9c02-69dbfe8c67a7" Jul 16 16:51:17.904450 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:17.904170 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2czkl" podUID="37b5e15a-de57-49d6-9c1a-fa42f89f8788" Jul 16 16:51:17.904450 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:17.904255 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hg5p" podUID="96a6eb58-3665-40df-a18f-4379194c172c" Jul 16 16:51:19.635511 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:19.635472 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs\") pod \"network-metrics-daemon-757wf\" (UID: \"2bd6202d-398f-4d16-9c02-69dbfe8c67a7\") " pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:19.635947 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:19.635526 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret\") pod \"global-pull-secret-syncer-6hg5p\" (UID: \"96a6eb58-3665-40df-a18f-4379194c172c\") " pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:19.635947 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:19.635609 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:19.635947 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:19.635682 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs podName:2bd6202d-398f-4d16-9c02-69dbfe8c67a7 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:35.635662226 +0000 UTC m=+33.154387463 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs") pod "network-metrics-daemon-757wf" (UID: "2bd6202d-398f-4d16-9c02-69dbfe8c67a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:19.635947 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:19.635693 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:19.635947 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:19.635759 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret podName:96a6eb58-3665-40df-a18f-4379194c172c nodeName:}" failed. No retries permitted until 2026-07-16 16:51:35.635738821 +0000 UTC m=+33.154464035 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret") pod "global-pull-secret-syncer-6hg5p" (UID: "96a6eb58-3665-40df-a18f-4379194c172c") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:19.736020 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:19.735970 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vc4n\" (UniqueName: \"kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n\") pod \"network-check-target-2czkl\" (UID: \"37b5e15a-de57-49d6-9c1a-fa42f89f8788\") " pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:19.736233 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:19.736175 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:51:19.736233 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:19.736205 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:51:19.736233 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:19.736219 2575 projected.go:196] Error preparing data for projected volume kube-api-access-5vc4n for pod openshift-network-diagnostics/network-check-target-2czkl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:19.736383 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:19.736282 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n podName:37b5e15a-de57-49d6-9c1a-fa42f89f8788 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:35.736261289 +0000 UTC m=+33.254986502 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vc4n" (UniqueName: "kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n") pod "network-check-target-2czkl" (UID: "37b5e15a-de57-49d6-9c1a-fa42f89f8788") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:19.903844 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:19.903753 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:19.903844 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:19.903789 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:19.903844 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:19.903789 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:19.904067 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:19.903909 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-757wf" podUID="2bd6202d-398f-4d16-9c02-69dbfe8c67a7" Jul 16 16:51:19.904067 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:19.903964 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hg5p" podUID="96a6eb58-3665-40df-a18f-4379194c172c" Jul 16 16:51:19.904067 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:19.904016 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2czkl" podUID="37b5e15a-de57-49d6-9c1a-fa42f89f8788" Jul 16 16:51:21.903470 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:21.903430 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:21.903470 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:21.903466 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:21.904024 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:21.903475 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:21.904024 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:21.903551 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2czkl" podUID="37b5e15a-de57-49d6-9c1a-fa42f89f8788" Jul 16 16:51:21.904024 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:21.903596 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hg5p" podUID="96a6eb58-3665-40df-a18f-4379194c172c" Jul 16 16:51:21.904024 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:21.903686 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-757wf" podUID="2bd6202d-398f-4d16-9c02-69dbfe8c67a7" Jul 16 16:51:22.995668 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:22.995198 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal" event={"ID":"b6563477dbb75145720caa9d8c4e6e99","Type":"ContainerStarted","Data":"10eccca1d34dc5dc2fcfafaa9aac64baef85afc653fd8de7774eb115c5673f81"} Jul 16 16:51:22.999002 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:22.997600 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-129.ec2.internal" event={"ID":"c90abb462952f54ad05f349a2dfb0ab3","Type":"ContainerStarted","Data":"74bbac826e7f5dcbbb1b1df2a02acd932c5758380bf5c36d5a0aa50dde1c4a47"} Jul 16 16:51:22.999785 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:22.999534 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6blcn" event={"ID":"f47ef2b6-4f33-4cee-9157-117ef5dffc52","Type":"ContainerStarted","Data":"99c9c4e2f5f00c6c9e7aa52443cf71c22f9747aed3e618248c6868e883961104"} Jul 16 16:51:23.003865 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.003323 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qvqn5" event={"ID":"bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989","Type":"ContainerStarted","Data":"f3328e09360b5c817b910a669bcee3ca3832a4487ace8b489e1ef21fc3bec162"} Jul 16 16:51:23.006241 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.006206 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ntxgv" event={"ID":"90c59dbd-833b-4fd1-9219-0677d51516ec","Type":"ContainerStarted","Data":"53ac7e5413395564eec64efa14cd89bbf878af0f95468792d17557cfd0de381a"} Jul 16 16:51:23.008224 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.008199 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bgc7n" event={"ID":"11336a91-8f10-40c6-943e-9383f8082593","Type":"ContainerStarted","Data":"36f4a053de691862a7f52adaf8acd6c956bab51e04a618853368f04a690fafae"} Jul 16 16:51:23.010320 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.010032 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-chkfd" event={"ID":"c111c675-66c3-42dc-8f46-74644b5907a3","Type":"ContainerStarted","Data":"cd76b54b5e88696d558baf8a760134af38c0174afba1e4d00bc70be57af03b0d"} Jul 16 16:51:23.012193 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.012175 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 16:51:23.012953 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.012924 2575 generic.go:356] "Generic (PLEG): container finished" podID="6ed467d2-8e31-4fc1-8631-3972722764e6" containerID="d57a8219c7a62f559d32b1099bdd5228ef4f2b597fda9d919d07de7b5b406fd7" exitCode=1 Jul 16 16:51:23.013042 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.013014 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" event={"ID":"6ed467d2-8e31-4fc1-8631-3972722764e6","Type":"ContainerStarted","Data":"56a3578aa2b468213fe2762be95ec21032cb8e046995058a7121387ddb5b250f"} Jul 16 16:51:23.013117 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.013080 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" event={"ID":"6ed467d2-8e31-4fc1-8631-3972722764e6","Type":"ContainerDied","Data":"d57a8219c7a62f559d32b1099bdd5228ef4f2b597fda9d919d07de7b5b406fd7"} Jul 16 16:51:23.013177 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.013142 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" event={"ID":"6ed467d2-8e31-4fc1-8631-3972722764e6","Type":"ContainerStarted","Data":"b2e87e12c0a2a176aedb0403d9e31edf75cb193d36b02b766b8b9ee6375a9a34"} Jul 16 16:51:23.014663 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.014638 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" event={"ID":"abcbbffd-bca1-417b-b445-81215aaaf8bb","Type":"ContainerStarted","Data":"b630cf8a636296f83fd8eccd73038086578a672c6d445e7b3c1a3bad4fb6a732"} Jul 16 16:51:23.017056 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.016562 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" event={"ID":"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f","Type":"ContainerStarted","Data":"553c61854717f3e8565c30dd666767028e6467bc49871419a290e875ab008b13"} Jul 16 16:51:23.072495 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.072350 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bgc7n" podStartSLOduration=1.7724547830000001 podStartE2EDuration="20.072335308s" totalImagesPullingTime="18.299880525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:23.037797612 +0000 UTC m=+20.556522846" watchObservedRunningTime="2026-07-16 16:51:23.072335308 +0000 UTC m=+20.591060536" Jul 16 16:51:23.093240 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.093204 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-chkfd" podStartSLOduration=5.897674728 podStartE2EDuration="20.093190819s" totalImagesPullingTime="14.195516091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:23.092198547 +0000 UTC m=+20.610923792" watchObservedRunningTime="2026-07-16 16:51:23.093190819 +0000 UTC m=+20.611916052" Jul 16 16:51:23.119332 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.119288 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ntxgv" podStartSLOduration=1.6311847369999999 podStartE2EDuration="20.119269139s" totalImagesPullingTime="18.488084402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:23.118878655 +0000 UTC m=+20.637603892" watchObservedRunningTime="2026-07-16 16:51:23.119269139 +0000 UTC m=+20.637994376" Jul 16 16:51:23.146781 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.146725 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fcbqn" podStartSLOduration=1.801600776 podStartE2EDuration="20.146705903s" totalImagesPullingTime="18.345105127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:23.14645572 +0000 UTC m=+20.665180956" watchObservedRunningTime="2026-07-16 16:51:23.146705903 +0000 UTC m=+20.665431139" Jul 16 16:51:23.170698 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.170647 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qvqn5" podStartSLOduration=5.694710282 podStartE2EDuration="20.170628336s" totalImagesPullingTime="14.475918054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:23.169907638 +0000 UTC m=+20.688632872" watchObservedRunningTime="2026-07-16 16:51:23.170628336 +0000 UTC m=+20.689353569" Jul 16 16:51:23.831413 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.831373 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:23.903839 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.903757 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:23.903997 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.903761 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:23.903997 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:23.903909 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-757wf" podUID="2bd6202d-398f-4d16-9c02-69dbfe8c67a7" Jul 16 16:51:23.903997 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.903941 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:23.904143 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:23.904061 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hg5p" podUID="96a6eb58-3665-40df-a18f-4379194c172c" Jul 16 16:51:23.904204 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:23.904176 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2czkl" podUID="37b5e15a-de57-49d6-9c1a-fa42f89f8788" Jul 16 16:51:24.000007 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:23.999970 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:24.020192 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:24.020158 2575 generic.go:356] "Generic (PLEG): container finished" podID="b6563477dbb75145720caa9d8c4e6e99" containerID="10eccca1d34dc5dc2fcfafaa9aac64baef85afc653fd8de7774eb115c5673f81" exitCode=0 Jul 16 16:51:24.020343 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:24.020241 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal" event={"ID":"b6563477dbb75145720caa9d8c4e6e99","Type":"ContainerDied","Data":"10eccca1d34dc5dc2fcfafaa9aac64baef85afc653fd8de7774eb115c5673f81"} Jul 16 16:51:24.021908 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:24.021889 2575 generic.go:356] "Generic (PLEG): container finished" podID="f47ef2b6-4f33-4cee-9157-117ef5dffc52" containerID="99c9c4e2f5f00c6c9e7aa52443cf71c22f9747aed3e618248c6868e883961104" exitCode=0 Jul 16 16:51:24.021999 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:24.021975 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6blcn" event={"ID":"f47ef2b6-4f33-4cee-9157-117ef5dffc52","Type":"ContainerDied","Data":"99c9c4e2f5f00c6c9e7aa52443cf71c22f9747aed3e618248c6868e883961104"} Jul 16 16:51:24.025431 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:24.025410 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 16:51:24.025919 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:24.025898 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" event={"ID":"6ed467d2-8e31-4fc1-8631-3972722764e6","Type":"ContainerStarted","Data":"bfb15ec679d7945f3504ea053c2fab27ebcb377126a291762cabe9d2cba7c20a"} Jul 16 16:51:24.026015 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:24.025971 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" event={"ID":"6ed467d2-8e31-4fc1-8631-3972722764e6","Type":"ContainerStarted","Data":"d9b762d55e1fbec29260f6eff29a458a3f1a7b017b75a60588fb613bc4cdd0df"} Jul 16 16:51:24.026085 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:24.026027 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" event={"ID":"6ed467d2-8e31-4fc1-8631-3972722764e6","Type":"ContainerStarted","Data":"aaf74b6a0c6180a1fdab9430be0cb01856014e5222c57c4bda42227bbdab38b0"} Jul 16 16:51:24.473920 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:24.473899 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-chkfd" Jul 16 16:51:24.474530 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:24.474507 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-chkfd" Jul 16 16:51:24.539515 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:24.539487 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 16:51:24.870115 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:24.870004 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T16:51:24.53950627Z","UUID":"988d9ea1-48bf-4fb9-abb2-f802c26fa3ec","Handler":null,"Name":"","Endpoint":""} Jul 16 16:51:24.872250 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:24.872226 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 16:51:24.872373 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:24.872276 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 16:51:25.029538 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:25.029501 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p4mk9" event={"ID":"19677c47-199e-4898-8f9d-b38ed966d3b4","Type":"ContainerStarted","Data":"93fbb0f30a5c27929649668fd472d2b007c2e562dd1c5df8a5bfc2a7861e3bb1"} Jul 16 16:51:25.032560 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:25.032529 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" event={"ID":"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f","Type":"ContainerStarted","Data":"c7acfa6f206ee3a21489f36b0a09f2742d8ea0916da4c6a16f4e9e2bbec7dd90"} Jul 16 16:51:25.043988 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:25.043954 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal" event={"ID":"b6563477dbb75145720caa9d8c4e6e99","Type":"ContainerStarted","Data":"5a3f43ac5b082905ce75ad9a812fb43536e964a82c3d683c172843748f893dee"} Jul 16 16:51:25.046021 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:25.045987 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-129.ec2.internal" event={"ID":"c90abb462952f54ad05f349a2dfb0ab3","Type":"ContainerStarted","Data":"d1d60c8ed790553d6aba776c06e7d70db22839c2a4523197fcdbb1a4fd915fba"} Jul 16 16:51:25.056400 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:25.056350 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-p4mk9" podStartSLOduration=12.152246432 podStartE2EDuration="22.056334125s" totalImagesPullingTime="9.904087693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:25.056228423 +0000 UTC m=+22.574953658" watchObservedRunningTime="2026-07-16 16:51:25.056334125 +0000 UTC m=+22.575059360" Jul 16 16:51:25.072863 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:25.072819 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-129.ec2.internal" podStartSLOduration=22.072804788 podStartE2EDuration="22.072804788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:25.072709849 +0000 UTC m=+22.591435086" watchObservedRunningTime="2026-07-16 16:51:25.072804788 +0000 UTC m=+22.591530022" Jul 16 16:51:25.088164 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:25.088116 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-129.ec2.internal" podStartSLOduration=22.088089708 podStartE2EDuration="22.088089708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:25.087824473 +0000 UTC m=+22.606549708" watchObservedRunningTime="2026-07-16 16:51:25.088089708 +0000 UTC m=+22.606814985" Jul 16 16:51:25.229422 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:25.229340 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:25.440832 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:25.440780 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:25.903970 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:25.903938 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:25.904210 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:25.903981 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:25.904210 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:25.904042 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:25.904210 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:25.904061 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hg5p" podUID="96a6eb58-3665-40df-a18f-4379194c172c" Jul 16 16:51:25.904210 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:25.904158 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-757wf" podUID="2bd6202d-398f-4d16-9c02-69dbfe8c67a7" Jul 16 16:51:25.904462 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:25.904217 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2czkl" podUID="37b5e15a-de57-49d6-9c1a-fa42f89f8788" Jul 16 16:51:25.963938 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:25.963908 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:26.050192 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:26.050158 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" event={"ID":"88bf4ea2-a95a-4fe5-ad54-39972ed01f6f","Type":"ContainerStarted","Data":"2a58a7703abe3b08a5947e28d73ceb1dc178be4f0ac8b213eb08ddaecb548459"} Jul 16 16:51:26.053760 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:26.053736 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 16:51:26.054146 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:26.054126 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" event={"ID":"6ed467d2-8e31-4fc1-8631-3972722764e6","Type":"ContainerStarted","Data":"709d3fb3b5d712b1782e6ee5c3c5498a7c69dd5bc7a55f254cb253dd6e38b146"} Jul 16 16:51:26.054263 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:26.054160 2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 16 16:51:26.072409 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:26.072361 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzkpw" podStartSLOduration=2.38863095 podStartE2EDuration="23.072347077s" totalImagesPullingTime="20.683716127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:26.071800589 +0000 UTC m=+23.590525824" watchObservedRunningTime="2026-07-16 16:51:26.072347077 +0000 UTC m=+23.591072510" Jul 16 16:51:26.108650 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:26.108619 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:26.190975 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:26.190887 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-chkfd" Jul 16 16:51:26.191484 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:26.191466 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-chkfd" Jul 16 16:51:26.288063 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:26.288020 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:27.903490 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:27.903296 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:27.903904 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:27.903301 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:27.903904 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:27.903588 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-757wf" podUID="2bd6202d-398f-4d16-9c02-69dbfe8c67a7" Jul 16 16:51:27.903904 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:27.903301 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:27.903904 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:27.903642 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hg5p" podUID="96a6eb58-3665-40df-a18f-4379194c172c" Jul 16 16:51:27.903904 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:27.903745 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2czkl" podUID="37b5e15a-de57-49d6-9c1a-fa42f89f8788" Jul 16 16:51:27.998588 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:27.998560 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:28.100429 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:28.100377 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:29.064711 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:29.064540 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 16:51:29.065458 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:29.065076 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" event={"ID":"6ed467d2-8e31-4fc1-8631-3972722764e6","Type":"ContainerStarted","Data":"06e4314e584729beae7576e2d3e9b348eba37e257dc82014cded0e2b5e8c39d9"} Jul 16 16:51:29.065458 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:29.065321 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:29.065458 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:29.065391 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:29.065458 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:29.065439 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:29.065586 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:29.065556 2575 scope.go:117] "RemoveContainer" containerID="d57a8219c7a62f559d32b1099bdd5228ef4f2b597fda9d919d07de7b5b406fd7" Jul 16 16:51:29.067262 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:29.067237 2575 generic.go:356] "Generic (PLEG): container finished" podID="f47ef2b6-4f33-4cee-9157-117ef5dffc52" containerID="625efb76c4ef537d8c6a92ec23ee28f9b1de25754b8b50e14a7bcd5fc297b99c" exitCode=0 Jul 16 16:51:29.067400 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:29.067272 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6blcn" event={"ID":"f47ef2b6-4f33-4cee-9157-117ef5dffc52","Type":"ContainerDied","Data":"625efb76c4ef537d8c6a92ec23ee28f9b1de25754b8b50e14a7bcd5fc297b99c"} Jul 16 16:51:29.081460 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:29.081441 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:29.081573 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:29.081560 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:51:29.904159 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:29.904127 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:29.904159 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:29.904159 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:29.904404 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:29.904128 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:29.904404 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:29.904261 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2czkl" podUID="37b5e15a-de57-49d6-9c1a-fa42f89f8788" Jul 16 16:51:29.904404 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:29.904323 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hg5p" podUID="96a6eb58-3665-40df-a18f-4379194c172c" Jul 16 16:51:29.907756 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:29.904902 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-757wf" podUID="2bd6202d-398f-4d16-9c02-69dbfe8c67a7" Jul 16 16:51:30.072660 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:30.072629 2575 generic.go:356] "Generic (PLEG): container finished" podID="f47ef2b6-4f33-4cee-9157-117ef5dffc52" containerID="b8316186b04fc9caeea1a71a5b2ccea55ea8f007fb908a3d5faf261ea7741b5a" exitCode=0 Jul 16 16:51:30.072936 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:30.072744 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6blcn" event={"ID":"f47ef2b6-4f33-4cee-9157-117ef5dffc52","Type":"ContainerDied","Data":"b8316186b04fc9caeea1a71a5b2ccea55ea8f007fb908a3d5faf261ea7741b5a"} Jul 16 16:51:30.077838 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:30.077816 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 16:51:30.078246 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:30.078214 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" event={"ID":"6ed467d2-8e31-4fc1-8631-3972722764e6","Type":"ContainerStarted","Data":"3572560453b1bcfc958f25285de50abd3c07e835004bc25a7dc5ffd86f845afd"} Jul 16 16:51:30.130925 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:30.130887 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" podStartSLOduration=8.987373484 podStartE2EDuration="27.130874s" totalImagesPullingTime="18.143500516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:30.129423141 +0000 UTC m=+27.648148386" watchObservedRunningTime="2026-07-16 16:51:30.130874 +0000 UTC m=+27.649599266" Jul 16 16:51:30.276807 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:30.276739 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-757wf"] Jul 16 16:51:30.276936 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:30.276876 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:30.277031 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:30.277005 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-757wf" podUID="2bd6202d-398f-4d16-9c02-69dbfe8c67a7" Jul 16 16:51:30.279843 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:30.279817 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2czkl"] Jul 16 16:51:30.279989 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:30.279923 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:30.280052 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:30.280010 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2czkl" podUID="37b5e15a-de57-49d6-9c1a-fa42f89f8788" Jul 16 16:51:30.280421 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:30.280398 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6hg5p"] Jul 16 16:51:30.280504 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:30.280486 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:30.280588 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:30.280568 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hg5p" podUID="96a6eb58-3665-40df-a18f-4379194c172c" Jul 16 16:51:31.083605 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:31.083522 2575 generic.go:356] "Generic (PLEG): container finished" podID="f47ef2b6-4f33-4cee-9157-117ef5dffc52" containerID="3677b3816e70e15d35ae3fa7047eb2332ec1fc27f0e9652031b63ec77f1b536a" exitCode=0 Jul 16 16:51:31.084147 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:31.083604 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6blcn" event={"ID":"f47ef2b6-4f33-4cee-9157-117ef5dffc52","Type":"ContainerDied","Data":"3677b3816e70e15d35ae3fa7047eb2332ec1fc27f0e9652031b63ec77f1b536a"} Jul 16 16:51:31.903359 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:31.903194 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:31.903482 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:31.903201 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:31.903482 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:31.903429 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hg5p" podUID="96a6eb58-3665-40df-a18f-4379194c172c" Jul 16 16:51:31.903567 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:31.903247 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:31.903567 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:31.903507 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-757wf" podUID="2bd6202d-398f-4d16-9c02-69dbfe8c67a7" Jul 16 16:51:31.903633 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:31.903591 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2czkl" podUID="37b5e15a-de57-49d6-9c1a-fa42f89f8788" Jul 16 16:51:32.181573 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:32.181536 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:32.316818 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:32.316784 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:33.197147 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:33.197116 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:33.306431 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:33.306399 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:33.413139 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:33.413057 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:33.903973 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:33.903944 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:33.903973 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:33.903967 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:33.903973 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:33.903949 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:33.904240 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:33.904080 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-757wf" podUID="2bd6202d-398f-4d16-9c02-69dbfe8c67a7" Jul 16 16:51:33.904240 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:33.904171 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hg5p" podUID="96a6eb58-3665-40df-a18f-4379194c172c" Jul 16 16:51:33.904339 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:33.904249 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2czkl" podUID="37b5e15a-de57-49d6-9c1a-fa42f89f8788" Jul 16 16:51:34.248423 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:34.248333 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:34.338238 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:34.338202 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:34.566433 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:34.566355 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:34.665439 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:34.665411 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:35.203538 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.203505 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:35.325522 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.325494 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-129.ec2.internal" event="NodeReady" Jul 16 16:51:35.325999 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.325630 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 16:51:35.382061 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.382027 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-789ddd76d-bbxxt"] Jul 16 16:51:35.405716 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.405643 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-t7p4b"] Jul 16 16:51:35.405859 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.405753 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.408791 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.408769 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 16:51:35.408904 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.408879 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2wxqg\"" Jul 16 16:51:35.408950 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.408901 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 16:51:35.409015 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.409000 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 16:51:35.416232 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.416215 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 16:51:35.431673 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.431655 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bzwp5"] Jul 16 16:51:35.431789 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.431773 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t7p4b" Jul 16 16:51:35.434642 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.434596 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 16:51:35.434775 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.434684 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 16:51:35.434775 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.434720 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 16:51:35.434875 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.434789 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lnc6v\"" Jul 16 16:51:35.447936 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.447918 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-789ddd76d-bbxxt"] Jul 16 16:51:35.448027 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.447961 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t7p4b"] Jul 16 16:51:35.448027 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.447977 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bzwp5"] Jul 16 16:51:35.448027 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.447922 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bzwp5" Jul 16 16:51:35.450826 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.450810 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 16:51:35.450915 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.450844 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xtf2c\"" Jul 16 16:51:35.450915 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.450862 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 16:51:35.557829 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.557799 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c64d5338-cf37-4a5f-b064-6b090cd3cde8-image-registry-private-configuration\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.557829 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.557830 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c64d5338-cf37-4a5f-b064-6b090cd3cde8-ca-trust-extracted\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.558016 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.557845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfp6g\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-kube-api-access-dfp6g\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.558016 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.557867 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbmrb\" (UniqueName: \"kubernetes.io/projected/cf501ec7-3530-4abf-bb14-e1f7aff7c53c-kube-api-access-gbmrb\") pod \"ingress-canary-t7p4b\" (UID: \"cf501ec7-3530-4abf-bb14-e1f7aff7c53c\") " pod="openshift-ingress-canary/ingress-canary-t7p4b" Jul 16 16:51:35.558016 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.557913 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls\") pod \"dns-default-bzwp5\" (UID: \"34ee6e6b-9608-4c5b-b556-1676477fe609\") " pod="openshift-dns/dns-default-bzwp5" Jul 16 16:51:35.558016 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.557964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-certificates\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.558016 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.557985 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c64d5338-cf37-4a5f-b064-6b090cd3cde8-trusted-ca\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.558016 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.558000 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34ee6e6b-9608-4c5b-b556-1676477fe609-config-volume\") pod \"dns-default-bzwp5\" (UID: \"34ee6e6b-9608-4c5b-b556-1676477fe609\") " pod="openshift-dns/dns-default-bzwp5" Jul 16 16:51:35.558336 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.558039 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34ee6e6b-9608-4c5b-b556-1676477fe609-tmp-dir\") pod \"dns-default-bzwp5\" (UID: \"34ee6e6b-9608-4c5b-b556-1676477fe609\") " pod="openshift-dns/dns-default-bzwp5" Jul 16 16:51:35.558336 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.558143 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf501ec7-3530-4abf-bb14-e1f7aff7c53c-cert\") pod \"ingress-canary-t7p4b\" (UID: \"cf501ec7-3530-4abf-bb14-e1f7aff7c53c\") " pod="openshift-ingress-canary/ingress-canary-t7p4b" Jul 16 16:51:35.558336 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.558198 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.558336 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.558227 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c64d5338-cf37-4a5f-b064-6b090cd3cde8-installation-pull-secrets\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.558336 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.558266 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vvwh\" (UniqueName: \"kubernetes.io/projected/34ee6e6b-9608-4c5b-b556-1676477fe609-kube-api-access-8vvwh\") pod \"dns-default-bzwp5\" (UID: \"34ee6e6b-9608-4c5b-b556-1676477fe609\") " pod="openshift-dns/dns-default-bzwp5" Jul 16 16:51:35.558336 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.558295 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-bound-sa-token\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.573477 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.573452 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:35.659193 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.659146 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-certificates\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.659193 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.659169 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c64d5338-cf37-4a5f-b064-6b090cd3cde8-trusted-ca\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.659193 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.659187 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34ee6e6b-9608-4c5b-b556-1676477fe609-config-volume\") pod \"dns-default-bzwp5\" (UID: \"34ee6e6b-9608-4c5b-b556-1676477fe609\") " pod="openshift-dns/dns-default-bzwp5" Jul 16 16:51:35.659405 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.659206 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34ee6e6b-9608-4c5b-b556-1676477fe609-tmp-dir\") pod \"dns-default-bzwp5\" (UID: \"34ee6e6b-9608-4c5b-b556-1676477fe609\") " pod="openshift-dns/dns-default-bzwp5" Jul 16 16:51:35.659405 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.659226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs\") pod \"network-metrics-daemon-757wf\" (UID: \"2bd6202d-398f-4d16-9c02-69dbfe8c67a7\") " pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:35.659405 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.659241 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf501ec7-3530-4abf-bb14-e1f7aff7c53c-cert\") pod \"ingress-canary-t7p4b\" (UID: \"cf501ec7-3530-4abf-bb14-e1f7aff7c53c\") " pod="openshift-ingress-canary/ingress-canary-t7p4b" Jul 16 16:51:35.659405 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:35.659346 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:35.659588 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:35.659414 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs podName:2bd6202d-398f-4d16-9c02-69dbfe8c67a7 nodeName:}" failed. No retries permitted until 2026-07-16 16:52:07.659395908 +0000 UTC m=+65.178121119 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs") pod "network-metrics-daemon-757wf" (UID: "2bd6202d-398f-4d16-9c02-69dbfe8c67a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:51:35.659685 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.659665 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34ee6e6b-9608-4c5b-b556-1676477fe609-tmp-dir\") pod \"dns-default-bzwp5\" (UID: \"34ee6e6b-9608-4c5b-b556-1676477fe609\") " pod="openshift-dns/dns-default-bzwp5" Jul 16 16:51:35.659782 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.659764 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret\") pod \"global-pull-secret-syncer-6hg5p\" (UID: \"96a6eb58-3665-40df-a18f-4379194c172c\") " pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:35.659833 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.659801 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.659884 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.659826 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c64d5338-cf37-4a5f-b064-6b090cd3cde8-installation-pull-secrets\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.659884 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.659846 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-certificates\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.659884 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.659872 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34ee6e6b-9608-4c5b-b556-1676477fe609-config-volume\") pod \"dns-default-bzwp5\" (UID: \"34ee6e6b-9608-4c5b-b556-1676477fe609\") " pod="openshift-dns/dns-default-bzwp5" Jul 16 16:51:35.660028 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:35.659915 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:35.660028 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:35.659943 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 16:51:35.660028 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:35.659961 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-789ddd76d-bbxxt: secret "image-registry-tls" not found Jul 16 16:51:35.660028 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:35.659977 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret podName:96a6eb58-3665-40df-a18f-4379194c172c nodeName:}" failed. No retries permitted until 2026-07-16 16:52:07.659962169 +0000 UTC m=+65.178687383 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret") pod "global-pull-secret-syncer-6hg5p" (UID: "96a6eb58-3665-40df-a18f-4379194c172c") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:51:35.660028 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.659998 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vvwh\" (UniqueName: \"kubernetes.io/projected/34ee6e6b-9608-4c5b-b556-1676477fe609-kube-api-access-8vvwh\") pod \"dns-default-bzwp5\" (UID: \"34ee6e6b-9608-4c5b-b556-1676477fe609\") " pod="openshift-dns/dns-default-bzwp5" Jul 16 16:51:35.660028 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:35.660008 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls podName:c64d5338-cf37-4a5f-b064-6b090cd3cde8 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:36.159991555 +0000 UTC m=+33.678716789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls") pod "image-registry-789ddd76d-bbxxt" (UID: "c64d5338-cf37-4a5f-b064-6b090cd3cde8") : secret "image-registry-tls" not found Jul 16 16:51:35.660333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.660043 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-bound-sa-token\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.660333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.660077 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c64d5338-cf37-4a5f-b064-6b090cd3cde8-image-registry-private-configuration\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.660333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.660123 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c64d5338-cf37-4a5f-b064-6b090cd3cde8-ca-trust-extracted\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.660333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.660150 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfp6g\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-kube-api-access-dfp6g\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.660333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.660179 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbmrb\" (UniqueName: \"kubernetes.io/projected/cf501ec7-3530-4abf-bb14-e1f7aff7c53c-kube-api-access-gbmrb\") pod \"ingress-canary-t7p4b\" (UID: \"cf501ec7-3530-4abf-bb14-e1f7aff7c53c\") " pod="openshift-ingress-canary/ingress-canary-t7p4b" Jul 16 16:51:35.660333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.660203 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls\") pod \"dns-default-bzwp5\" (UID: \"34ee6e6b-9608-4c5b-b556-1676477fe609\") " pod="openshift-dns/dns-default-bzwp5" Jul 16 16:51:35.660333 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:35.660315 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 16:51:35.660628 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:35.660353 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls podName:34ee6e6b-9608-4c5b-b556-1676477fe609 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:36.160339733 +0000 UTC m=+33.679064962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls") pod "dns-default-bzwp5" (UID: "34ee6e6b-9608-4c5b-b556-1676477fe609") : secret "dns-default-metrics-tls" not found Jul 16 16:51:35.660628 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.660350 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c64d5338-cf37-4a5f-b064-6b090cd3cde8-trusted-ca\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.660628 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.660514 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c64d5338-cf37-4a5f-b064-6b090cd3cde8-ca-trust-extracted\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.663513 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.663489 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c64d5338-cf37-4a5f-b064-6b090cd3cde8-installation-pull-secrets\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.663597 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.663500 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf501ec7-3530-4abf-bb14-e1f7aff7c53c-cert\") pod \"ingress-canary-t7p4b\" (UID: \"cf501ec7-3530-4abf-bb14-e1f7aff7c53c\") " pod="openshift-ingress-canary/ingress-canary-t7p4b" Jul 16 16:51:35.663597 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.663577 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c64d5338-cf37-4a5f-b064-6b090cd3cde8-image-registry-private-configuration\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.676340 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.676321 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-bound-sa-token\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.678532 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.678516 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfp6g\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-kube-api-access-dfp6g\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:35.682947 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.682927 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:35.683615 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.683594 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbmrb\" (UniqueName: \"kubernetes.io/projected/cf501ec7-3530-4abf-bb14-e1f7aff7c53c-kube-api-access-gbmrb\") pod \"ingress-canary-t7p4b\" (UID: \"cf501ec7-3530-4abf-bb14-e1f7aff7c53c\") " pod="openshift-ingress-canary/ingress-canary-t7p4b" Jul 16 16:51:35.685365 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.685347 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vvwh\" (UniqueName: \"kubernetes.io/projected/34ee6e6b-9608-4c5b-b556-1676477fe609-kube-api-access-8vvwh\") pod \"dns-default-bzwp5\" (UID: \"34ee6e6b-9608-4c5b-b556-1676477fe609\") " pod="openshift-dns/dns-default-bzwp5" Jul 16 16:51:35.739738 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.739719 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t7p4b" Jul 16 16:51:35.760782 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.760762 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vc4n\" (UniqueName: \"kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n\") pod \"network-check-target-2czkl\" (UID: \"37b5e15a-de57-49d6-9c1a-fa42f89f8788\") " pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:35.760918 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:35.760899 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:51:35.760918 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:35.760922 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:51:35.761079 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:35.760933 2575 projected.go:196] Error preparing data for projected volume kube-api-access-5vc4n for pod openshift-network-diagnostics/network-check-target-2czkl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:35.761079 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:35.760984 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n podName:37b5e15a-de57-49d6-9c1a-fa42f89f8788 nodeName:}" failed. No retries permitted until 2026-07-16 16:52:07.760969594 +0000 UTC m=+65.279694806 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5vc4n" (UniqueName: "kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n") pod "network-check-target-2czkl" (UID: "37b5e15a-de57-49d6-9c1a-fa42f89f8788") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:51:35.903624 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.903601 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:51:35.903740 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.903702 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:51:35.903740 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.903735 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:51:35.906524 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.906499 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 16:51:35.906624 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.906551 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h8r6p\"" Jul 16 16:51:35.906694 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.906623 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s2b72\"" Jul 16 16:51:35.906694 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.906555 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 16:51:35.906694 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.906551 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 16:51:35.906958 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:35.906941 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 16:51:36.047609 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:36.047454 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t7p4b"] Jul 16 16:51:36.050751 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:36.050714 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf501ec7_3530_4abf_bb14_e1f7aff7c53c.slice/crio-c7c70ebe976ca831421923630d7d993dcaefc1a108eb0aa612d903bfbbfa5aa4 WatchSource:0}: Error finding container c7c70ebe976ca831421923630d7d993dcaefc1a108eb0aa612d903bfbbfa5aa4: Status 404 returned error can't find the container with id c7c70ebe976ca831421923630d7d993dcaefc1a108eb0aa612d903bfbbfa5aa4 Jul 16 16:51:36.099153 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:36.099110 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6blcn" event={"ID":"f47ef2b6-4f33-4cee-9157-117ef5dffc52","Type":"ContainerStarted","Data":"645bed2888d306a165af4770d7c604f3fa9a788b8da2d2d259707e4570653fb5"} Jul 16 16:51:36.100324 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:36.100301 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t7p4b" event={"ID":"cf501ec7-3530-4abf-bb14-e1f7aff7c53c","Type":"ContainerStarted","Data":"c7c70ebe976ca831421923630d7d993dcaefc1a108eb0aa612d903bfbbfa5aa4"} Jul 16 16:51:36.165241 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:36.165162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls\") pod \"dns-default-bzwp5\" (UID: \"34ee6e6b-9608-4c5b-b556-1676477fe609\") " pod="openshift-dns/dns-default-bzwp5" Jul 16 16:51:36.165335 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:36.165240 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:36.165335 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:36.165308 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 16:51:36.165401 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:36.165341 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 16:51:36.165401 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:36.165351 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-789ddd76d-bbxxt: secret "image-registry-tls" not found Jul 16 16:51:36.165401 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:36.165371 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls podName:34ee6e6b-9608-4c5b-b556-1676477fe609 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:37.165354448 +0000 UTC m=+34.684079661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls") pod "dns-default-bzwp5" (UID: "34ee6e6b-9608-4c5b-b556-1676477fe609") : secret "dns-default-metrics-tls" not found Jul 16 16:51:36.165401 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:36.165388 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls podName:c64d5338-cf37-4a5f-b064-6b090cd3cde8 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:37.165377091 +0000 UTC m=+34.684102303 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls") pod "image-registry-789ddd76d-bbxxt" (UID: "c64d5338-cf37-4a5f-b064-6b090cd3cde8") : secret "image-registry-tls" not found Jul 16 16:51:37.105796 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:37.105764 2575 generic.go:356] "Generic (PLEG): container finished" podID="f47ef2b6-4f33-4cee-9157-117ef5dffc52" containerID="645bed2888d306a165af4770d7c604f3fa9a788b8da2d2d259707e4570653fb5" exitCode=0 Jul 16 16:51:37.106491 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:37.105814 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6blcn" event={"ID":"f47ef2b6-4f33-4cee-9157-117ef5dffc52","Type":"ContainerDied","Data":"645bed2888d306a165af4770d7c604f3fa9a788b8da2d2d259707e4570653fb5"} Jul 16 16:51:37.173219 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:37.173193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls\") pod \"dns-default-bzwp5\" (UID: \"34ee6e6b-9608-4c5b-b556-1676477fe609\") " pod="openshift-dns/dns-default-bzwp5" Jul 16 16:51:37.173363 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:37.173296 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:37.173429 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:37.173359 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 16:51:37.173429 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:37.173416 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 16:51:37.173429 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:37.173429 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-789ddd76d-bbxxt: secret "image-registry-tls" not found Jul 16 16:51:37.173585 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:37.173435 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls podName:34ee6e6b-9608-4c5b-b556-1676477fe609 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:39.17341366 +0000 UTC m=+36.692138874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls") pod "dns-default-bzwp5" (UID: "34ee6e6b-9608-4c5b-b556-1676477fe609") : secret "dns-default-metrics-tls" not found Jul 16 16:51:37.173585 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:37.173473 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls podName:c64d5338-cf37-4a5f-b064-6b090cd3cde8 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:39.173458428 +0000 UTC m=+36.692183641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls") pod "image-registry-789ddd76d-bbxxt" (UID: "c64d5338-cf37-4a5f-b064-6b090cd3cde8") : secret "image-registry-tls" not found Jul 16 16:51:37.186201 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:37.186174 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:37.313521 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:37.313488 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:38.111869 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:38.111838 2575 generic.go:356] "Generic (PLEG): container finished" podID="f47ef2b6-4f33-4cee-9157-117ef5dffc52" containerID="7583c87b7f4185349a46b616e694a37028bf6be0b892e9ab69e477382bd87bd3" exitCode=0 Jul 16 16:51:38.112462 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:38.111877 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6blcn" event={"ID":"f47ef2b6-4f33-4cee-9157-117ef5dffc52","Type":"ContainerDied","Data":"7583c87b7f4185349a46b616e694a37028bf6be0b892e9ab69e477382bd87bd3"} Jul 16 16:51:39.117666 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:39.117470 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6blcn" event={"ID":"f47ef2b6-4f33-4cee-9157-117ef5dffc52","Type":"ContainerStarted","Data":"891c58a465b806a8a40595c547598e44ea99ae30fddcdfa90d135794d5186ebd"} Jul 16 16:51:39.118952 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:39.118931 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t7p4b" event={"ID":"cf501ec7-3530-4abf-bb14-e1f7aff7c53c","Type":"ContainerStarted","Data":"754a9bee4475b1e48ca93cd6b409b03bd395704b012a9b1eda93c86c33b476c0"} Jul 16 16:51:39.144753 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:39.144710 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6blcn" podStartSLOduration=11.085590698 podStartE2EDuration="36.144695375s" totalImagesPullingTime="25.059104677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:39.144493588 +0000 UTC m=+36.663218826" watchObservedRunningTime="2026-07-16 16:51:39.144695375 +0000 UTC m=+36.663420608" Jul 16 16:51:39.162008 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:39.161968 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-t7p4b" podStartSLOduration=1.638592037 podStartE2EDuration="4.161955119s" totalImagesPullingTime="2.523363082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:39.161508245 +0000 UTC m=+36.680233480" watchObservedRunningTime="2026-07-16 16:51:39.161955119 +0000 UTC m=+36.680680582" Jul 16 16:51:39.189780 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:39.189753 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:39.189885 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:39.189816 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls\") pod \"dns-default-bzwp5\" (UID: \"34ee6e6b-9608-4c5b-b556-1676477fe609\") " pod="openshift-dns/dns-default-bzwp5" Jul 16 16:51:39.189885 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:39.189865 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 16:51:39.189885 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:39.189880 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-789ddd76d-bbxxt: secret "image-registry-tls" not found Jul 16 16:51:39.190007 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:39.189934 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls podName:c64d5338-cf37-4a5f-b064-6b090cd3cde8 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:43.189920563 +0000 UTC m=+40.708645787 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls") pod "image-registry-789ddd76d-bbxxt" (UID: "c64d5338-cf37-4a5f-b064-6b090cd3cde8") : secret "image-registry-tls" not found Jul 16 16:51:39.190007 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:39.189948 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 16:51:39.190007 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:39.189999 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls podName:34ee6e6b-9608-4c5b-b556-1676477fe609 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:43.189983281 +0000 UTC m=+40.708708494 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls") pod "dns-default-bzwp5" (UID: "34ee6e6b-9608-4c5b-b556-1676477fe609") : secret "dns-default-metrics-tls" not found Jul 16 16:51:39.307806 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:39.307737 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:39.435505 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:39.435438 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:41.073741 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.073712 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:41.082860 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.082832 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mxnxv"] Jul 16 16:51:41.103372 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.103352 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mxnxv"] Jul 16 16:51:41.103470 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.103446 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mxnxv" Jul 16 16:51:41.106043 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.106022 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 16 16:51:41.106043 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.106039 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4gcdd\"" Jul 16 16:51:41.107149 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.107134 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 16 16:51:41.174177 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.174113 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:41.202246 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.202220 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhcdg\" (UniqueName: \"kubernetes.io/projected/ae61bb97-d7ee-4bc0-803d-82a5ad11d73d-kube-api-access-qhcdg\") pod \"migrator-7f59b54f6c-mxnxv\" (UID: \"ae61bb97-d7ee-4bc0-803d-82a5ad11d73d\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mxnxv" Jul 16 16:51:41.302700 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.302675 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhcdg\" (UniqueName: \"kubernetes.io/projected/ae61bb97-d7ee-4bc0-803d-82a5ad11d73d-kube-api-access-qhcdg\") pod \"migrator-7f59b54f6c-mxnxv\" (UID: \"ae61bb97-d7ee-4bc0-803d-82a5ad11d73d\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mxnxv" Jul 16 16:51:41.314400 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.314369 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhcdg\" (UniqueName: \"kubernetes.io/projected/ae61bb97-d7ee-4bc0-803d-82a5ad11d73d-kube-api-access-qhcdg\") pod \"migrator-7f59b54f6c-mxnxv\" (UID: \"ae61bb97-d7ee-4bc0-803d-82a5ad11d73d\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mxnxv" Jul 16 16:51:41.346315 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.346244 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:41.412149 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.412120 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mxnxv" Jul 16 16:51:41.478705 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.478676 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:41.550590 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.550562 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mxnxv"] Jul 16 16:51:41.667714 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.667681 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xql75"] Jul 16 16:51:41.695941 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.695916 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xql75"] Jul 16 16:51:41.696065 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.696038 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:41.698853 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.698824 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rbpsf\"" Jul 16 16:51:41.698853 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.698851 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 16:51:41.699016 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.698863 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 16:51:41.699293 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.699281 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 16:51:41.700039 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.700021 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 16:51:41.719474 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.719448 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:41.805730 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.805693 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8c09583-4113-4f64-974f-03c3047db739-data-volume\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:41.805870 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.805752 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8c09583-4113-4f64-974f-03c3047db739-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:41.805870 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.805775 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:41.805870 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.805824 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e8c09583-4113-4f64-974f-03c3047db739-crio-socket\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:41.805970 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.805869 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rfzd\" (UniqueName: \"kubernetes.io/projected/e8c09583-4113-4f64-974f-03c3047db739-kube-api-access-8rfzd\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:41.882199 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.882168 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:41.906466 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.906443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rfzd\" (UniqueName: \"kubernetes.io/projected/e8c09583-4113-4f64-974f-03c3047db739-kube-api-access-8rfzd\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:41.906606 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.906482 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8c09583-4113-4f64-974f-03c3047db739-data-volume\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:41.906606 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.906535 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8c09583-4113-4f64-974f-03c3047db739-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:41.906606 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.906559 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:41.906752 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.906623 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e8c09583-4113-4f64-974f-03c3047db739-crio-socket\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:41.906752 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:41.906674 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 16:51:41.906752 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:41.906750 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e8c09583-4113-4f64-974f-03c3047db739-kube-rbac-proxy-cm podName:e8c09583-4113-4f64-974f-03c3047db739 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:42.406733486 +0000 UTC m=+39.925458712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e8c09583-4113-4f64-974f-03c3047db739-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xql75" (UID: "e8c09583-4113-4f64-974f-03c3047db739") : configmap "kube-rbac-proxy" not found Jul 16 16:51:41.906905 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:41.906775 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 16:51:41.906905 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:41.906851 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls podName:e8c09583-4113-4f64-974f-03c3047db739 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:42.406819939 +0000 UTC m=+39.925545168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xql75" (UID: "e8c09583-4113-4f64-974f-03c3047db739") : secret "insights-runtime-extractor-tls" not found Jul 16 16:51:41.906905 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.906893 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8c09583-4113-4f64-974f-03c3047db739-data-volume\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:41.906905 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.906896 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e8c09583-4113-4f64-974f-03c3047db739-crio-socket\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:41.918227 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:41.918172 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rfzd\" (UniqueName: \"kubernetes.io/projected/e8c09583-4113-4f64-974f-03c3047db739-kube-api-access-8rfzd\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:42.126714 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:42.126673 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mxnxv" event={"ID":"ae61bb97-d7ee-4bc0-803d-82a5ad11d73d","Type":"ContainerStarted","Data":"79561a68bd8cb23d1ded45f0692ed7bcd9bd662a0ddf7bfb0975f0e830bf5a35"} Jul 16 16:51:42.285349 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:42.285276 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:42.411312 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:42.411275 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8c09583-4113-4f64-974f-03c3047db739-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:42.411312 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:42.411311 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:42.411495 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:42.411407 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 16:51:42.411495 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:42.411453 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 16:51:42.411495 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:42.411473 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e8c09583-4113-4f64-974f-03c3047db739-kube-rbac-proxy-cm podName:e8c09583-4113-4f64-974f-03c3047db739 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:43.411454536 +0000 UTC m=+40.930179748 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e8c09583-4113-4f64-974f-03c3047db739-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xql75" (UID: "e8c09583-4113-4f64-974f-03c3047db739") : configmap "kube-rbac-proxy" not found Jul 16 16:51:42.411495 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:42.411490 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls podName:e8c09583-4113-4f64-974f-03c3047db739 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:43.411480265 +0000 UTC m=+40.930205481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xql75" (UID: "e8c09583-4113-4f64-974f-03c3047db739") : secret "insights-runtime-extractor-tls" not found Jul 16 16:51:43.217493 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:43.217460 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:43.217867 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:43.217507 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls\") pod \"dns-default-bzwp5\" (UID: \"34ee6e6b-9608-4c5b-b556-1676477fe609\") " pod="openshift-dns/dns-default-bzwp5" Jul 16 16:51:43.217867 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:43.217597 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 16:51:43.217867 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:43.217603 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 16:51:43.217867 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:43.217620 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-789ddd76d-bbxxt: secret "image-registry-tls" not found Jul 16 16:51:43.217867 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:43.217645 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls podName:34ee6e6b-9608-4c5b-b556-1676477fe609 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:51.217631505 +0000 UTC m=+48.736356718 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls") pod "dns-default-bzwp5" (UID: "34ee6e6b-9608-4c5b-b556-1676477fe609") : secret "dns-default-metrics-tls" not found Jul 16 16:51:43.217867 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:43.217665 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls podName:c64d5338-cf37-4a5f-b064-6b090cd3cde8 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:51.217650019 +0000 UTC m=+48.736375231 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls") pod "image-registry-789ddd76d-bbxxt" (UID: "c64d5338-cf37-4a5f-b064-6b090cd3cde8") : secret "image-registry-tls" not found Jul 16 16:51:43.418451 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:43.418417 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8c09583-4113-4f64-974f-03c3047db739-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:43.418451 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:43.418451 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:43.418687 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:43.418569 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 16:51:43.418687 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:43.418642 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 16:51:43.418687 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:43.418649 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e8c09583-4113-4f64-974f-03c3047db739-kube-rbac-proxy-cm podName:e8c09583-4113-4f64-974f-03c3047db739 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:45.418628691 +0000 UTC m=+42.937353915 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e8c09583-4113-4f64-974f-03c3047db739-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xql75" (UID: "e8c09583-4113-4f64-974f-03c3047db739") : configmap "kube-rbac-proxy" not found Jul 16 16:51:43.418844 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:43.418701 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls podName:e8c09583-4113-4f64-974f-03c3047db739 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:45.418687097 +0000 UTC m=+42.937412308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xql75" (UID: "e8c09583-4113-4f64-974f-03c3047db739") : secret "insights-runtime-extractor-tls" not found Jul 16 16:51:43.744064 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:43.744011 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-kxn4n"] Jul 16 16:51:43.761401 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:43.761380 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-kxn4n"] Jul 16 16:51:43.761498 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:43.761459 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-kxn4n" Jul 16 16:51:43.764582 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:43.764561 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 16 16:51:43.764690 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:43.764639 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 16 16:51:43.764690 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:43.764673 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 16 16:51:43.764789 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:43.764764 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 16 16:51:43.765014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:43.764995 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zjnr8\"" Jul 16 16:51:43.819837 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:43.819797 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:43.921541 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:43.921512 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49bxj\" (UniqueName: \"kubernetes.io/projected/9c2a6b34-6936-4ffd-91ca-e062e67d798a-kube-api-access-49bxj\") pod \"service-ca-79f68b9f4f-kxn4n\" (UID: \"9c2a6b34-6936-4ffd-91ca-e062e67d798a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-kxn4n" Jul 16 16:51:43.921662 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:43.921566 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9c2a6b34-6936-4ffd-91ca-e062e67d798a-signing-key\") pod \"service-ca-79f68b9f4f-kxn4n\" (UID: \"9c2a6b34-6936-4ffd-91ca-e062e67d798a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-kxn4n" Jul 16 16:51:43.921662 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:43.921646 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c2a6b34-6936-4ffd-91ca-e062e67d798a-signing-cabundle\") pod \"service-ca-79f68b9f4f-kxn4n\" (UID: \"9c2a6b34-6936-4ffd-91ca-e062e67d798a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-kxn4n" Jul 16 16:51:43.941788 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:43.941761 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:43.943983 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:43.943967 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bgc7n_11336a91-8f10-40c6-943e-9383f8082593/dns-node-resolver/0.log" Jul 16 16:51:44.022485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:44.022418 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c2a6b34-6936-4ffd-91ca-e062e67d798a-signing-cabundle\") pod \"service-ca-79f68b9f4f-kxn4n\" (UID: \"9c2a6b34-6936-4ffd-91ca-e062e67d798a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-kxn4n" Jul 16 16:51:44.022575 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:44.022494 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49bxj\" (UniqueName: \"kubernetes.io/projected/9c2a6b34-6936-4ffd-91ca-e062e67d798a-kube-api-access-49bxj\") pod \"service-ca-79f68b9f4f-kxn4n\" (UID: \"9c2a6b34-6936-4ffd-91ca-e062e67d798a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-kxn4n" Jul 16 16:51:44.022575 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:44.022543 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9c2a6b34-6936-4ffd-91ca-e062e67d798a-signing-key\") pod \"service-ca-79f68b9f4f-kxn4n\" (UID: \"9c2a6b34-6936-4ffd-91ca-e062e67d798a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-kxn4n" Jul 16 16:51:44.023112 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:44.023071 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c2a6b34-6936-4ffd-91ca-e062e67d798a-signing-cabundle\") pod \"service-ca-79f68b9f4f-kxn4n\" (UID: \"9c2a6b34-6936-4ffd-91ca-e062e67d798a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-kxn4n" Jul 16 16:51:44.024955 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:44.024927 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9c2a6b34-6936-4ffd-91ca-e062e67d798a-signing-key\") pod \"service-ca-79f68b9f4f-kxn4n\" (UID: \"9c2a6b34-6936-4ffd-91ca-e062e67d798a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-kxn4n" Jul 16 16:51:44.032425 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:44.032402 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49bxj\" (UniqueName: \"kubernetes.io/projected/9c2a6b34-6936-4ffd-91ca-e062e67d798a-kube-api-access-49bxj\") pod \"service-ca-79f68b9f4f-kxn4n\" (UID: \"9c2a6b34-6936-4ffd-91ca-e062e67d798a\") " pod="openshift-service-ca/service-ca-79f68b9f4f-kxn4n" Jul 16 16:51:44.070311 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:44.070290 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-kxn4n" Jul 16 16:51:44.134381 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:44.134131 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mxnxv" event={"ID":"ae61bb97-d7ee-4bc0-803d-82a5ad11d73d","Type":"ContainerStarted","Data":"8acaf713ff7e66c9cd7049c33e17f818c55895aef6f07b9427866db382b1dbdc"} Jul 16 16:51:44.134381 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:44.134193 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mxnxv" event={"ID":"ae61bb97-d7ee-4bc0-803d-82a5ad11d73d","Type":"ContainerStarted","Data":"36595a4b33cde962fae6061bfda68e3dbc7ceef2da25aabc79351a458a8279b4"} Jul 16 16:51:44.221956 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:44.221912 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-mxnxv" podStartSLOduration=1.306977089 podStartE2EDuration="3.221896285s" totalImagesPullingTime="1.914919196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:44.162381737 +0000 UTC m=+41.681106972" watchObservedRunningTime="2026-07-16 16:51:44.221896285 +0000 UTC m=+41.740621518" Jul 16 16:51:44.222767 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:44.222748 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-kxn4n"] Jul 16 16:51:44.225769 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:51:44.225735 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2a6b34_6936_4ffd_91ca_e062e67d798a.slice/crio-40240619e4de4350742f176636b1c8a718606972b592a8d7bbfe07cfdb8f8dba WatchSource:0}: Error finding container 40240619e4de4350742f176636b1c8a718606972b592a8d7bbfe07cfdb8f8dba: Status 404 returned error can't find the container with id 40240619e4de4350742f176636b1c8a718606972b592a8d7bbfe07cfdb8f8dba Jul 16 16:51:44.344259 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:44.344206 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:44.452734 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:44.452706 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:44.648427 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:44.648398 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:45.138554 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:45.138510 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-kxn4n" event={"ID":"9c2a6b34-6936-4ffd-91ca-e062e67d798a","Type":"ContainerStarted","Data":"40240619e4de4350742f176636b1c8a718606972b592a8d7bbfe07cfdb8f8dba"} Jul 16 16:51:45.345566 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:45.345542 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qvqn5_bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989/node-ca/0.log" Jul 16 16:51:45.433246 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:45.433170 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8c09583-4113-4f64-974f-03c3047db739-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:45.433246 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:45.433204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:45.433430 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:45.433290 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 16:51:45.433430 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:45.433318 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 16:51:45.433430 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:45.433353 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e8c09583-4113-4f64-974f-03c3047db739-kube-rbac-proxy-cm podName:e8c09583-4113-4f64-974f-03c3047db739 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:49.433338012 +0000 UTC m=+46.952063229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e8c09583-4113-4f64-974f-03c3047db739-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xql75" (UID: "e8c09583-4113-4f64-974f-03c3047db739") : configmap "kube-rbac-proxy" not found Jul 16 16:51:45.433430 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:45.433367 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls podName:e8c09583-4113-4f64-974f-03c3047db739 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:49.433361201 +0000 UTC m=+46.952086413 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xql75" (UID: "e8c09583-4113-4f64-974f-03c3047db739") : secret "insights-runtime-extractor-tls" not found Jul 16 16:51:45.745379 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:45.745308 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t7p4b_cf501ec7-3530-4abf-bb14-e1f7aff7c53c/serve-healthcheck-canary/0.log" Jul 16 16:51:46.144789 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:46.144754 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-mxnxv_ae61bb97-d7ee-4bc0-803d-82a5ad11d73d/migrator/0.log" Jul 16 16:51:46.347937 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:46.347912 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-mxnxv_ae61bb97-d7ee-4bc0-803d-82a5ad11d73d/graceful-termination/0.log" Jul 16 16:51:47.145057 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:47.145027 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-kxn4n" event={"ID":"9c2a6b34-6936-4ffd-91ca-e062e67d798a","Type":"ContainerStarted","Data":"f556a6b41ca1e29a1ce0d1d9616980a922f2ff4849a4f3a0c3cddf877e68c5eb"} Jul 16 16:51:47.164910 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:47.164866 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-kxn4n" podStartSLOduration=1.751981952 podStartE2EDuration="4.164852386s" totalImagesPullingTime="2.412870434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:47.164401891 +0000 UTC m=+44.683127128" watchObservedRunningTime="2026-07-16 16:51:47.164852386 +0000 UTC m=+44.683577619" Jul 16 16:51:48.759336 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:48.759298 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:48.889535 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:48.889493 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:49.014451 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:49.014373 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:49.129676 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:49.129644 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:49.238213 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:49.238179 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:49.462810 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:49.462777 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8c09583-4113-4f64-974f-03c3047db739-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:49.462810 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:49.462811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:49.463054 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:49.462905 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 16:51:49.463054 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:49.462912 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 16:51:49.463054 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:49.462958 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls podName:e8c09583-4113-4f64-974f-03c3047db739 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:57.462946071 +0000 UTC m=+54.981671282 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xql75" (UID: "e8c09583-4113-4f64-974f-03c3047db739") : secret "insights-runtime-extractor-tls" not found Jul 16 16:51:49.463054 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:49.462971 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e8c09583-4113-4f64-974f-03c3047db739-kube-rbac-proxy-cm podName:e8c09583-4113-4f64-974f-03c3047db739 nodeName:}" failed. No retries permitted until 2026-07-16 16:51:57.462965521 +0000 UTC m=+54.981690733 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e8c09583-4113-4f64-974f-03c3047db739-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xql75" (UID: "e8c09583-4113-4f64-974f-03c3047db739") : configmap "kube-rbac-proxy" not found Jul 16 16:51:51.275721 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:51.275689 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:51:51.276128 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:51.275740 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls\") pod \"dns-default-bzwp5\" (UID: \"34ee6e6b-9608-4c5b-b556-1676477fe609\") " pod="openshift-dns/dns-default-bzwp5" Jul 16 16:51:51.276128 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:51.275848 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 16:51:51.276128 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:51.275865 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-789ddd76d-bbxxt: secret "image-registry-tls" not found Jul 16 16:51:51.276128 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:51.275868 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 16:51:51.276128 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:51.275921 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls podName:34ee6e6b-9608-4c5b-b556-1676477fe609 nodeName:}" failed. No retries permitted until 2026-07-16 16:52:07.275905135 +0000 UTC m=+64.794630346 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls") pod "dns-default-bzwp5" (UID: "34ee6e6b-9608-4c5b-b556-1676477fe609") : secret "dns-default-metrics-tls" not found Jul 16 16:51:51.276128 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:51.275936 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls podName:c64d5338-cf37-4a5f-b064-6b090cd3cde8 nodeName:}" failed. No retries permitted until 2026-07-16 16:52:07.275929858 +0000 UTC m=+64.794655070 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls") pod "image-registry-789ddd76d-bbxxt" (UID: "c64d5338-cf37-4a5f-b064-6b090cd3cde8") : secret "image-registry-tls" not found Jul 16 16:51:51.414311 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:51.414282 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:51.553129 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:51.553035 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:51.810963 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:51.810879 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:52.256061 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:52.256030 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:52.407980 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:52.407950 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:52.517001 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:52.516926 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:52.723730 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:52.723701 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:52.822902 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:52.822823 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:53.190820 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:53.190793 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:53.294738 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:53.294711 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:53.782382 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:53.782352 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:54.177552 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:54.177523 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:54.581963 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:54.581887 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:54.797130 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:54.797087 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:55.018001 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:55.017973 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:55.119183 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:55.119149 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:55.217052 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:55.217017 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:55.791027 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:55.790995 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:55.955833 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:55.955807 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:56.055913 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:56.055843 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:56.369936 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:56.369902 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:56.609941 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:56.609910 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:56.979052 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:56.979017 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:57.188220 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:57.188175 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:57.525662 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:57.525621 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8c09583-4113-4f64-974f-03c3047db739-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:57.525825 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:57.525669 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:57.525890 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:57.525870 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 16:51:57.525967 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:51:57.525958 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls podName:e8c09583-4113-4f64-974f-03c3047db739 nodeName:}" failed. No retries permitted until 2026-07-16 16:52:13.525940483 +0000 UTC m=+71.044665714 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xql75" (UID: "e8c09583-4113-4f64-974f-03c3047db739") : secret "insights-runtime-extractor-tls" not found Jul 16 16:51:57.526173 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:57.526158 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8c09583-4113-4f64-974f-03c3047db739-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:51:57.582705 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:57.582674 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:58.171722 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:58.171670 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:58.786704 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:58.786636 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:58.977899 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:58.977867 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:59.179584 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:59.179545 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:51:59.387136 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:51:59.387085 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:00.028956 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:00.028928 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:01.102547 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:01.102519 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fj6nb" Jul 16 16:52:01.547879 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:01.547795 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:07.300563 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.300523 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:52:07.301005 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.300573 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls\") pod \"dns-default-bzwp5\" (UID: \"34ee6e6b-9608-4c5b-b556-1676477fe609\") " pod="openshift-dns/dns-default-bzwp5" Jul 16 16:52:07.302896 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.302876 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls\") pod \"image-registry-789ddd76d-bbxxt\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:52:07.302958 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.302897 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34ee6e6b-9608-4c5b-b556-1676477fe609-metrics-tls\") pod \"dns-default-bzwp5\" (UID: \"34ee6e6b-9608-4c5b-b556-1676477fe609\") " pod="openshift-dns/dns-default-bzwp5" Jul 16 16:52:07.519850 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.519822 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2wxqg\"" Jul 16 16:52:07.526172 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.526155 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:52:07.561906 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.561841 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xtf2c\"" Jul 16 16:52:07.566911 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.566893 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bzwp5" Jul 16 16:52:07.704713 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.704681 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs\") pod \"network-metrics-daemon-757wf\" (UID: \"2bd6202d-398f-4d16-9c02-69dbfe8c67a7\") " pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:52:07.704840 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.704735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret\") pod \"global-pull-secret-syncer-6hg5p\" (UID: \"96a6eb58-3665-40df-a18f-4379194c172c\") " pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:52:07.707618 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.707593 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 16:52:07.707754 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.707645 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 16:52:07.717434 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.717406 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96a6eb58-3665-40df-a18f-4379194c172c-original-pull-secret\") pod \"global-pull-secret-syncer-6hg5p\" (UID: \"96a6eb58-3665-40df-a18f-4379194c172c\") " pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:52:07.717537 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.717507 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd6202d-398f-4d16-9c02-69dbfe8c67a7-metrics-certs\") pod \"network-metrics-daemon-757wf\" (UID: \"2bd6202d-398f-4d16-9c02-69dbfe8c67a7\") " pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:52:07.722964 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.722939 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-789ddd76d-bbxxt"] Jul 16 16:52:07.726236 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:52:07.726213 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64d5338_cf37_4a5f_b064_6b090cd3cde8.slice/crio-1e8a616b43a7edb17ecd4eed9670705957461b2a9954366e574c4cefdb85f5f2 WatchSource:0}: Error finding container 1e8a616b43a7edb17ecd4eed9670705957461b2a9954366e574c4cefdb85f5f2: Status 404 returned error can't find the container with id 1e8a616b43a7edb17ecd4eed9670705957461b2a9954366e574c4cefdb85f5f2 Jul 16 16:52:07.727712 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.727695 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s2b72\"" Jul 16 16:52:07.732191 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:52:07.732169 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ee6e6b_9608_4c5b_b556_1676477fe609.slice/crio-bca6e9eb5070f13e1f0b05b8669485a87d891c146676a5bf1540898a4257f23f WatchSource:0}: Error finding container bca6e9eb5070f13e1f0b05b8669485a87d891c146676a5bf1540898a4257f23f: Status 404 returned error can't find the container with id bca6e9eb5070f13e1f0b05b8669485a87d891c146676a5bf1540898a4257f23f Jul 16 16:52:07.734296 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.734269 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bzwp5"] Jul 16 16:52:07.735498 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.735484 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-757wf" Jul 16 16:52:07.772304 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.772285 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-qvvhc"] Jul 16 16:52:07.791964 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.791941 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2"] Jul 16 16:52:07.792074 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.792048 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-qvvhc" Jul 16 16:52:07.796531 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.796343 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-h42zw\"" Jul 16 16:52:07.797293 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.796874 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 16 16:52:07.797293 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.797143 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 16 16:52:07.805475 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.805274 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vc4n\" (UniqueName: \"kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n\") pod \"network-check-target-2czkl\" (UID: \"37b5e15a-de57-49d6-9c1a-fa42f89f8788\") " pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:52:07.805475 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.805327 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/24e57db4-9ef2-4f43-bb8d-36d2774ac52b-nginx-conf\") pod \"networking-console-plugin-6cf856758c-qvvhc\" (UID: \"24e57db4-9ef2-4f43-bb8d-36d2774ac52b\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-qvvhc" Jul 16 16:52:07.805475 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.805361 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24e57db4-9ef2-4f43-bb8d-36d2774ac52b-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-qvvhc\" (UID: \"24e57db4-9ef2-4f43-bb8d-36d2774ac52b\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-qvvhc" Jul 16 16:52:07.812535 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.812482 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 16:52:07.816294 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.816265 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz"] Jul 16 16:52:07.816711 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.816688 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2" Jul 16 16:52:07.821814 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.821794 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 16 16:52:07.822257 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.822239 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 16 16:52:07.822407 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.822385 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 16 16:52:07.822479 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.822327 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 16:52:07.823120 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.823086 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 16 16:52:07.823221 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.823152 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xbjn5\"" Jul 16 16:52:07.829202 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.829159 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vc4n\" (UniqueName: \"kubernetes.io/projected/37b5e15a-de57-49d6-9c1a-fa42f89f8788-kube-api-access-5vc4n\") pod \"network-check-target-2czkl\" (UID: \"37b5e15a-de57-49d6-9c1a-fa42f89f8788\") " pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:52:07.838072 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.838054 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-qvvhc"] Jul 16 16:52:07.838183 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.838115 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2"] Jul 16 16:52:07.838183 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.838141 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-789ddd76d-bbxxt"] Jul 16 16:52:07.838183 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.838162 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz"] Jul 16 16:52:07.838183 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.838059 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:07.841421 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.841400 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 16 16:52:07.841530 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.841466 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 16 16:52:07.841530 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.841502 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 16 16:52:07.841845 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.841832 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 16 16:52:07.903895 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.903747 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-757wf"] Jul 16 16:52:07.906059 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.906036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9cgh\" (UniqueName: \"kubernetes.io/projected/ebdabf05-6ccd-48da-a4cf-9f24b3c90423-kube-api-access-w9cgh\") pod \"managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2\" (UID: \"ebdabf05-6ccd-48da-a4cf-9f24b3c90423\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2" Jul 16 16:52:07.906193 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.906067 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/16142617-2cfb-4bbe-acf5-73ff08d12339-ca\") pod \"cluster-proxy-proxy-agent-6fb86554db-dhrjz\" (UID: \"16142617-2cfb-4bbe-acf5-73ff08d12339\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:07.906193 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.906114 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/24e57db4-9ef2-4f43-bb8d-36d2774ac52b-nginx-conf\") pod \"networking-console-plugin-6cf856758c-qvvhc\" (UID: \"24e57db4-9ef2-4f43-bb8d-36d2774ac52b\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-qvvhc" Jul 16 16:52:07.906193 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.906184 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/16142617-2cfb-4bbe-acf5-73ff08d12339-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fb86554db-dhrjz\" (UID: \"16142617-2cfb-4bbe-acf5-73ff08d12339\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:07.906326 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:52:07.906219 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd6202d_398f_4d16_9c02_69dbfe8c67a7.slice/crio-0a4b91f653687718a890b10bf6bcd9b679f496dabbb89b9ba05179dc9f7c25d9 WatchSource:0}: Error finding container 0a4b91f653687718a890b10bf6bcd9b679f496dabbb89b9ba05179dc9f7c25d9: Status 404 returned error can't find the container with id 0a4b91f653687718a890b10bf6bcd9b679f496dabbb89b9ba05179dc9f7c25d9 Jul 16 16:52:07.906326 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.906281 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndhbp\" (UniqueName: \"kubernetes.io/projected/16142617-2cfb-4bbe-acf5-73ff08d12339-kube-api-access-ndhbp\") pod \"cluster-proxy-proxy-agent-6fb86554db-dhrjz\" (UID: \"16142617-2cfb-4bbe-acf5-73ff08d12339\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:07.906389 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.906356 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/16142617-2cfb-4bbe-acf5-73ff08d12339-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fb86554db-dhrjz\" (UID: \"16142617-2cfb-4bbe-acf5-73ff08d12339\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:07.906429 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.906397 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/16142617-2cfb-4bbe-acf5-73ff08d12339-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fb86554db-dhrjz\" (UID: \"16142617-2cfb-4bbe-acf5-73ff08d12339\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:07.906462 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.906451 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24e57db4-9ef2-4f43-bb8d-36d2774ac52b-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-qvvhc\" (UID: \"24e57db4-9ef2-4f43-bb8d-36d2774ac52b\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-qvvhc" Jul 16 16:52:07.906510 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.906476 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ebdabf05-6ccd-48da-a4cf-9f24b3c90423-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2\" (UID: \"ebdabf05-6ccd-48da-a4cf-9f24b3c90423\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2" Jul 16 16:52:07.906571 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.906548 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/16142617-2cfb-4bbe-acf5-73ff08d12339-hub\") pod \"cluster-proxy-proxy-agent-6fb86554db-dhrjz\" (UID: \"16142617-2cfb-4bbe-acf5-73ff08d12339\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:07.906801 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.906779 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/24e57db4-9ef2-4f43-bb8d-36d2774ac52b-nginx-conf\") pod \"networking-console-plugin-6cf856758c-qvvhc\" (UID: \"24e57db4-9ef2-4f43-bb8d-36d2774ac52b\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-qvvhc" Jul 16 16:52:07.908822 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.908805 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/24e57db4-9ef2-4f43-bb8d-36d2774ac52b-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-qvvhc\" (UID: \"24e57db4-9ef2-4f43-bb8d-36d2774ac52b\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-qvvhc" Jul 16 16:52:07.915112 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.915077 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-644d85d8c9-qphqg"] Jul 16 16:52:07.929643 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.929624 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:07.937824 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.937805 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-644d85d8c9-qphqg"] Jul 16 16:52:07.991816 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:07.991767 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:08.007402 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.007377 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxzmf\" (UniqueName: \"kubernetes.io/projected/bfb8d84a-c5c9-465f-969b-626b6187f3a9-kube-api-access-bxzmf\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.007553 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.007412 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/16142617-2cfb-4bbe-acf5-73ff08d12339-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fb86554db-dhrjz\" (UID: \"16142617-2cfb-4bbe-acf5-73ff08d12339\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:08.007553 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.007445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ebdabf05-6ccd-48da-a4cf-9f24b3c90423-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2\" (UID: \"ebdabf05-6ccd-48da-a4cf-9f24b3c90423\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2" Jul 16 16:52:08.007553 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.007467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/16142617-2cfb-4bbe-acf5-73ff08d12339-hub\") pod \"cluster-proxy-proxy-agent-6fb86554db-dhrjz\" (UID: \"16142617-2cfb-4bbe-acf5-73ff08d12339\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:08.007553 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.007488 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9cgh\" (UniqueName: \"kubernetes.io/projected/ebdabf05-6ccd-48da-a4cf-9f24b3c90423-kube-api-access-w9cgh\") pod \"managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2\" (UID: \"ebdabf05-6ccd-48da-a4cf-9f24b3c90423\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2" Jul 16 16:52:08.007553 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.007510 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/16142617-2cfb-4bbe-acf5-73ff08d12339-ca\") pod \"cluster-proxy-proxy-agent-6fb86554db-dhrjz\" (UID: \"16142617-2cfb-4bbe-acf5-73ff08d12339\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:08.007553 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.007531 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfb8d84a-c5c9-465f-969b-626b6187f3a9-bound-sa-token\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.007892 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.007570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/16142617-2cfb-4bbe-acf5-73ff08d12339-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fb86554db-dhrjz\" (UID: \"16142617-2cfb-4bbe-acf5-73ff08d12339\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:08.007892 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.007596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndhbp\" (UniqueName: \"kubernetes.io/projected/16142617-2cfb-4bbe-acf5-73ff08d12339-kube-api-access-ndhbp\") pod \"cluster-proxy-proxy-agent-6fb86554db-dhrjz\" (UID: \"16142617-2cfb-4bbe-acf5-73ff08d12339\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:08.007892 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.007622 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bfb8d84a-c5c9-465f-969b-626b6187f3a9-ca-trust-extracted\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.007892 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.007644 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfb8d84a-c5c9-465f-969b-626b6187f3a9-trusted-ca\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.007892 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.007668 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bfb8d84a-c5c9-465f-969b-626b6187f3a9-registry-certificates\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.007892 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.007691 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bfb8d84a-c5c9-465f-969b-626b6187f3a9-image-registry-private-configuration\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.007892 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.007714 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bfb8d84a-c5c9-465f-969b-626b6187f3a9-installation-pull-secrets\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.007892 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.007760 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/16142617-2cfb-4bbe-acf5-73ff08d12339-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fb86554db-dhrjz\" (UID: \"16142617-2cfb-4bbe-acf5-73ff08d12339\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:08.007892 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.007781 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfb8d84a-c5c9-465f-969b-626b6187f3a9-registry-tls\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.008354 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.008220 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/16142617-2cfb-4bbe-acf5-73ff08d12339-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fb86554db-dhrjz\" (UID: \"16142617-2cfb-4bbe-acf5-73ff08d12339\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:08.010325 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.010297 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/16142617-2cfb-4bbe-acf5-73ff08d12339-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fb86554db-dhrjz\" (UID: \"16142617-2cfb-4bbe-acf5-73ff08d12339\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:08.010457 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.010356 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/16142617-2cfb-4bbe-acf5-73ff08d12339-hub\") pod \"cluster-proxy-proxy-agent-6fb86554db-dhrjz\" (UID: \"16142617-2cfb-4bbe-acf5-73ff08d12339\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:08.010457 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.010444 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/16142617-2cfb-4bbe-acf5-73ff08d12339-ca\") pod \"cluster-proxy-proxy-agent-6fb86554db-dhrjz\" (UID: \"16142617-2cfb-4bbe-acf5-73ff08d12339\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:08.010698 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.010677 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ebdabf05-6ccd-48da-a4cf-9f24b3c90423-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2\" (UID: \"ebdabf05-6ccd-48da-a4cf-9f24b3c90423\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2" Jul 16 16:52:08.010735 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.010686 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/16142617-2cfb-4bbe-acf5-73ff08d12339-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fb86554db-dhrjz\" (UID: \"16142617-2cfb-4bbe-acf5-73ff08d12339\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:08.014368 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.014352 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hg5p" Jul 16 16:52:08.018874 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.018854 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9cgh\" (UniqueName: \"kubernetes.io/projected/ebdabf05-6ccd-48da-a4cf-9f24b3c90423-kube-api-access-w9cgh\") pod \"managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2\" (UID: \"ebdabf05-6ccd-48da-a4cf-9f24b3c90423\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2" Jul 16 16:52:08.019168 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.019151 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndhbp\" (UniqueName: \"kubernetes.io/projected/16142617-2cfb-4bbe-acf5-73ff08d12339-kube-api-access-ndhbp\") pod \"cluster-proxy-proxy-agent-6fb86554db-dhrjz\" (UID: \"16142617-2cfb-4bbe-acf5-73ff08d12339\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:08.023744 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.023726 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h8r6p\"" Jul 16 16:52:08.031990 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.031975 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:52:08.108543 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.108470 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfb8d84a-c5c9-465f-969b-626b6187f3a9-trusted-ca\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.108543 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.108516 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bfb8d84a-c5c9-465f-969b-626b6187f3a9-registry-certificates\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.108727 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.108546 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bfb8d84a-c5c9-465f-969b-626b6187f3a9-image-registry-private-configuration\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.108727 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.108570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bfb8d84a-c5c9-465f-969b-626b6187f3a9-installation-pull-secrets\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.108727 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.108620 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfb8d84a-c5c9-465f-969b-626b6187f3a9-registry-tls\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.108727 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.108655 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxzmf\" (UniqueName: \"kubernetes.io/projected/bfb8d84a-c5c9-465f-969b-626b6187f3a9-kube-api-access-bxzmf\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.108976 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.108742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfb8d84a-c5c9-465f-969b-626b6187f3a9-bound-sa-token\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.108976 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.108792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bfb8d84a-c5c9-465f-969b-626b6187f3a9-ca-trust-extracted\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.109462 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.109252 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bfb8d84a-c5c9-465f-969b-626b6187f3a9-ca-trust-extracted\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.111794 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.110418 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bfb8d84a-c5c9-465f-969b-626b6187f3a9-registry-certificates\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.111794 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.111454 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfb8d84a-c5c9-465f-969b-626b6187f3a9-trusted-ca\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.112066 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.111983 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-qvvhc" Jul 16 16:52:08.112379 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.112352 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfb8d84a-c5c9-465f-969b-626b6187f3a9-registry-tls\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.113020 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.112972 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bfb8d84a-c5c9-465f-969b-626b6187f3a9-installation-pull-secrets\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.114677 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.114626 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bfb8d84a-c5c9-465f-969b-626b6187f3a9-image-registry-private-configuration\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.122918 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.122884 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfb8d84a-c5c9-465f-969b-626b6187f3a9-bound-sa-token\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.123438 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.123404 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxzmf\" (UniqueName: \"kubernetes.io/projected/bfb8d84a-c5c9-465f-969b-626b6187f3a9-kube-api-access-bxzmf\") pod \"image-registry-644d85d8c9-qphqg\" (UID: \"bfb8d84a-c5c9-465f-969b-626b6187f3a9\") " pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.136967 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.136861 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2" Jul 16 16:52:08.147368 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.147166 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" Jul 16 16:52:08.187515 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.187430 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:08.202133 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.201316 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" event={"ID":"c64d5338-cf37-4a5f-b064-6b090cd3cde8","Type":"ContainerStarted","Data":"3555ff3f7a5467c91b18900adea8b3cd45d3f2cff6b4137a05f85186553cbaad"} Jul 16 16:52:08.202133 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.201411 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" event={"ID":"c64d5338-cf37-4a5f-b064-6b090cd3cde8","Type":"ContainerStarted","Data":"1e8a616b43a7edb17ecd4eed9670705957461b2a9954366e574c4cefdb85f5f2"} Jul 16 16:52:08.202133 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.201677 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:52:08.204999 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.204918 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-757wf" event={"ID":"2bd6202d-398f-4d16-9c02-69dbfe8c67a7","Type":"ContainerStarted","Data":"0a4b91f653687718a890b10bf6bcd9b679f496dabbb89b9ba05179dc9f7c25d9"} Jul 16 16:52:08.209403 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.209346 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bzwp5" event={"ID":"34ee6e6b-9608-4c5b-b556-1676477fe609","Type":"ContainerStarted","Data":"bca6e9eb5070f13e1f0b05b8669485a87d891c146676a5bf1540898a4257f23f"} Jul 16 16:52:08.216164 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.214034 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6hg5p"] Jul 16 16:52:08.219039 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:52:08.219019 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a6eb58_3665_40df_a18f_4379194c172c.slice/crio-773a8ac30b88b15effe7caba12c03f3f8d3e36f7ad5bb9fa0c78617d6ac5ad4b WatchSource:0}: Error finding container 773a8ac30b88b15effe7caba12c03f3f8d3e36f7ad5bb9fa0c78617d6ac5ad4b: Status 404 returned error can't find the container with id 773a8ac30b88b15effe7caba12c03f3f8d3e36f7ad5bb9fa0c78617d6ac5ad4b Jul 16 16:52:08.239222 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.239043 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2czkl"] Jul 16 16:52:08.252716 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.252389 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:08.294559 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.294530 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:08.321471 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.320342 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" podStartSLOduration=34.320317853 podStartE2EDuration="34.320317853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:08.245472778 +0000 UTC m=+65.764198013" watchObservedRunningTime="2026-07-16 16:52:08.320317853 +0000 UTC m=+65.839043088" Jul 16 16:52:08.321471 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.321341 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-qvvhc"] Jul 16 16:52:08.340332 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:52:08.340293 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e57db4_9ef2_4f43_bb8d_36d2774ac52b.slice/crio-2a4c8491d75cc67d700f1f5cb5f1652d9040935a612230d05a305ec9a42a6f46 WatchSource:0}: Error finding container 2a4c8491d75cc67d700f1f5cb5f1652d9040935a612230d05a305ec9a42a6f46: Status 404 returned error can't find the container with id 2a4c8491d75cc67d700f1f5cb5f1652d9040935a612230d05a305ec9a42a6f46 Jul 16 16:52:08.361714 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.361653 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz"] Jul 16 16:52:08.366208 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:52:08.366172 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16142617_2cfb_4bbe_acf5_73ff08d12339.slice/crio-2ed87e51221c7a12518790a50d5013168a9cb6537eadacc4cd84178651f406f0 WatchSource:0}: Error finding container 2ed87e51221c7a12518790a50d5013168a9cb6537eadacc4cd84178651f406f0: Status 404 returned error can't find the container with id 2ed87e51221c7a12518790a50d5013168a9cb6537eadacc4cd84178651f406f0 Jul 16 16:52:08.377436 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.377362 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2"] Jul 16 16:52:08.383761 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:52:08.383718 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebdabf05_6ccd_48da_a4cf_9f24b3c90423.slice/crio-eb56eae1d3fffd2a34fc1ca99ef3be925c5de606b3ea2e89a75893675601c464 WatchSource:0}: Error finding container eb56eae1d3fffd2a34fc1ca99ef3be925c5de606b3ea2e89a75893675601c464: Status 404 returned error can't find the container with id eb56eae1d3fffd2a34fc1ca99ef3be925c5de606b3ea2e89a75893675601c464 Jul 16 16:52:08.471493 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.471462 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-644d85d8c9-qphqg"] Jul 16 16:52:08.475573 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:52:08.475542 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb8d84a_c5c9_465f_969b_626b6187f3a9.slice/crio-7a075b297eeb751a12395d48a6f07f04661b02639b93f1662b02d746b35bb0df WatchSource:0}: Error finding container 7a075b297eeb751a12395d48a6f07f04661b02639b93f1662b02d746b35bb0df: Status 404 returned error can't find the container with id 7a075b297eeb751a12395d48a6f07f04661b02639b93f1662b02d746b35bb0df Jul 16 16:52:08.708820 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.708731 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:08.829164 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.829059 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:08.987534 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:08.987438 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:09.218961 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:09.218873 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-qvvhc" event={"ID":"24e57db4-9ef2-4f43-bb8d-36d2774ac52b","Type":"ContainerStarted","Data":"2a4c8491d75cc67d700f1f5cb5f1652d9040935a612230d05a305ec9a42a6f46"} Jul 16 16:52:09.222081 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:09.222035 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6hg5p" event={"ID":"96a6eb58-3665-40df-a18f-4379194c172c","Type":"ContainerStarted","Data":"773a8ac30b88b15effe7caba12c03f3f8d3e36f7ad5bb9fa0c78617d6ac5ad4b"} Jul 16 16:52:09.226613 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:09.226473 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" event={"ID":"bfb8d84a-c5c9-465f-969b-626b6187f3a9","Type":"ContainerStarted","Data":"0220197e12d36a6aa8c02baa775fb01cd83d91e1e279defcac264e7b8604449c"} Jul 16 16:52:09.227779 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:09.227687 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:09.227779 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:09.227725 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" event={"ID":"bfb8d84a-c5c9-465f-969b-626b6187f3a9","Type":"ContainerStarted","Data":"7a075b297eeb751a12395d48a6f07f04661b02639b93f1662b02d746b35bb0df"} Jul 16 16:52:09.230372 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:09.230298 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" event={"ID":"16142617-2cfb-4bbe-acf5-73ff08d12339","Type":"ContainerStarted","Data":"2ed87e51221c7a12518790a50d5013168a9cb6537eadacc4cd84178651f406f0"} Jul 16 16:52:09.233158 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:09.233083 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2" event={"ID":"ebdabf05-6ccd-48da-a4cf-9f24b3c90423","Type":"ContainerStarted","Data":"eb56eae1d3fffd2a34fc1ca99ef3be925c5de606b3ea2e89a75893675601c464"} Jul 16 16:52:09.235161 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:09.235085 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2czkl" event={"ID":"37b5e15a-de57-49d6-9c1a-fa42f89f8788","Type":"ContainerStarted","Data":"d922fbfa6301c5451e568f29b0494df391a6e9efc187075c4c75852ca65cf618"} Jul 16 16:52:09.251000 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:09.250909 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" podStartSLOduration=2.250887707 podStartE2EDuration="2.250887707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:09.24819464 +0000 UTC m=+66.766919878" watchObservedRunningTime="2026-07-16 16:52:09.250887707 +0000 UTC m=+66.769612970" Jul 16 16:52:09.288595 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:09.288507 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:10.375572 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:10.375471 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:10.562050 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:10.561947 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:13.566862 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:13.566785 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:52:13.569445 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:13.569420 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8c09583-4113-4f64-974f-03c3047db739-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xql75\" (UID: \"e8c09583-4113-4f64-974f-03c3047db739\") " pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:52:13.808132 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:13.808086 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rbpsf\"" Jul 16 16:52:13.815633 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:13.815603 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xql75" Jul 16 16:52:15.396613 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:15.396581 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:15.511639 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:15.511606 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:16.041388 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:16.041357 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:16.146088 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:16.146059 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:16.279081 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:16.279014 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:16.477805 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:16.477683 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xql75"] Jul 16 16:52:17.274446 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.274408 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-qvvhc" event={"ID":"24e57db4-9ef2-4f43-bb8d-36d2774ac52b","Type":"ContainerStarted","Data":"a3c2b71b8c608d11e10333980039e32a67d1c8a3660146b12cb2caab6804f9ee"} Jul 16 16:52:17.278002 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.276879 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6hg5p" event={"ID":"96a6eb58-3665-40df-a18f-4379194c172c","Type":"ContainerStarted","Data":"de2a2bf64e3eef0364b4112da6c84b4f425a579d0cb8b2b46f7204020ed58289"} Jul 16 16:52:17.280056 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.279991 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xql75" event={"ID":"e8c09583-4113-4f64-974f-03c3047db739","Type":"ContainerStarted","Data":"b0674180b56501df3f015273579c1e82c9fd00f3727181a5b0c7dde077c2e822"} Jul 16 16:52:17.283504 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.283479 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-757wf" event={"ID":"2bd6202d-398f-4d16-9c02-69dbfe8c67a7","Type":"ContainerStarted","Data":"ac4486bb27e09c4dd18155bc63f11e48cf607f334f868707542db536a88b4525"} Jul 16 16:52:17.283613 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.283526 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-757wf" event={"ID":"2bd6202d-398f-4d16-9c02-69dbfe8c67a7","Type":"ContainerStarted","Data":"6cc74a56e3c53dfb59727ada13d6a157303c604562833bfb02db7a33951804c4"} Jul 16 16:52:17.286977 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.286545 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bzwp5" event={"ID":"34ee6e6b-9608-4c5b-b556-1676477fe609","Type":"ContainerStarted","Data":"1c1487731188d42680c7204e2c478139d56934b3b34c76caedf38bb1f7db06a6"} Jul 16 16:52:17.286977 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.286601 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bzwp5" Jul 16 16:52:17.286977 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.286626 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bzwp5" event={"ID":"34ee6e6b-9608-4c5b-b556-1676477fe609","Type":"ContainerStarted","Data":"65d9b0d0c2b2891b425c8669b187f520182a9c9941c7a9189aab1efa1465ffee"} Jul 16 16:52:17.289388 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.289364 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" event={"ID":"16142617-2cfb-4bbe-acf5-73ff08d12339","Type":"ContainerStarted","Data":"df297de1e3dfa8f58a1b8aa241b8d5fd394d353a27b45f468e883859a3ac74c2"} Jul 16 16:52:17.289491 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.289403 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" event={"ID":"16142617-2cfb-4bbe-acf5-73ff08d12339","Type":"ContainerStarted","Data":"930c36188ba255797177ed7aad401eef1554233599c44acf2a552e8a643df173"} Jul 16 16:52:17.289491 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.289421 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" event={"ID":"16142617-2cfb-4bbe-acf5-73ff08d12339","Type":"ContainerStarted","Data":"2fc4e1c38001c1d7c57d1e5f2b16427e9ccb3ea63d34564504b3eb467820022a"} Jul 16 16:52:17.291422 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.291399 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2" event={"ID":"ebdabf05-6ccd-48da-a4cf-9f24b3c90423","Type":"ContainerStarted","Data":"af6a616ede7fb24fce58f17bdfb26157579c52a8db78f4657bb6637427ddd7e4"} Jul 16 16:52:17.293621 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.293590 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2czkl" event={"ID":"37b5e15a-de57-49d6-9c1a-fa42f89f8788","Type":"ContainerStarted","Data":"6757c5236066ca314bf98e5fb0eb5480ec31f3c11a2b64239ef2c6e94b9b473b"} Jul 16 16:52:17.294677 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.294000 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:52:17.295861 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.295820 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-qvvhc" podStartSLOduration=2.310913701 podStartE2EDuration="10.295804174s" totalImagesPullingTime="7.984890473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:17.294298449 +0000 UTC m=+74.813023684" watchObservedRunningTime="2026-07-16 16:52:17.295804174 +0000 UTC m=+74.814529390" Jul 16 16:52:17.316026 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.315949 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bzwp5" podStartSLOduration=33.998648697 podStartE2EDuration="42.31588443s" totalImagesPullingTime="8.317235733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:17.315424452 +0000 UTC m=+74.834149688" watchObservedRunningTime="2026-07-16 16:52:17.31588443 +0000 UTC m=+74.834609670" Jul 16 16:52:17.333122 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.333034 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2czkl" podStartSLOduration=66.254853498 podStartE2EDuration="1m14.333016596s" totalImagesPullingTime="8.078163098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:17.331821257 +0000 UTC m=+74.850546492" watchObservedRunningTime="2026-07-16 16:52:17.333016596 +0000 UTC m=+74.851741831" Jul 16 16:52:17.350824 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.350775 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc8f556d8-dhcg2" podStartSLOduration=2.405738349 podStartE2EDuration="10.350757722s" totalImagesPullingTime="7.945019373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:17.349492896 +0000 UTC m=+74.868218130" watchObservedRunningTime="2026-07-16 16:52:17.350757722 +0000 UTC m=+74.869482947" Jul 16 16:52:17.374657 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.374607 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb86554db-dhrjz" podStartSLOduration=2.411879298 podStartE2EDuration="10.374589891s" totalImagesPullingTime="7.962710593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:17.373531253 +0000 UTC m=+74.892256489" watchObservedRunningTime="2026-07-16 16:52:17.374589891 +0000 UTC m=+74.893315127" Jul 16 16:52:17.387604 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.387575 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:17.395423 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.395378 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-757wf" podStartSLOduration=65.905160042 podStartE2EDuration="1m14.395365457s" totalImagesPullingTime="8.490205415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:17.395249727 +0000 UTC m=+74.913974962" watchObservedRunningTime="2026-07-16 16:52:17.395365457 +0000 UTC m=+74.914090690" Jul 16 16:52:17.422111 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.413967 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6hg5p" podStartSLOduration=66.302399719 podStartE2EDuration="1m14.413952975s" totalImagesPullingTime="8.111553256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:17.412961822 +0000 UTC m=+74.931687057" watchObservedRunningTime="2026-07-16 16:52:17.413952975 +0000 UTC m=+74.932678209" Jul 16 16:52:17.547561 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:17.547480 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:18.877688 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:18.877654 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:19.013145 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:19.013113 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:19.299429 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:19.299292 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:19.437706 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:19.437677 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:19.624451 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:19.624412 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:19.966613 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:19.966537 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:20.097944 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:20.097916 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:20.389250 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:20.389217 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:20.516676 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:20.516649 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:20.669270 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:20.669196 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:21.180045 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:21.180011 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:21.411506 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:21.411473 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:21.709315 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:21.709244 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:22.319029 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:22.318992 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xql75" event={"ID":"e8c09583-4113-4f64-974f-03c3047db739","Type":"ContainerStarted","Data":"f2657fb67de1fef25d018101a2c168ee8a78cd5bd519440837586ccaac64eb4a"} Jul 16 16:52:22.405019 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:22.404982 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:22.491751 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:22.491717 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:24.749419 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:24.749379 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:24.854146 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:24.854014 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:25.332214 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:25.332179 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xql75" event={"ID":"e8c09583-4113-4f64-974f-03c3047db739","Type":"ContainerStarted","Data":"10d5fe4815f5ea04dc8d8e1d7b22bf49885b3049dbf083fe035200842b4b12d1"} Jul 16 16:52:26.403748 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:26.403712 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:26.556532 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:26.556494 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:27.300499 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:27.300472 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bzwp5" Jul 16 16:52:28.183472 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.183433 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:28.207977 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.207951 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:52:28.316452 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.316425 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:28.427405 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.427381 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:28.470954 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.470896 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-95brm"] Jul 16 16:52:28.474137 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.474119 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.477568 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.477549 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 16:52:28.477687 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.477588 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 16:52:28.477884 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.477749 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 16:52:28.477884 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.477594 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 16:52:28.478299 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.478281 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2tl75\"" Jul 16 16:52:28.478603 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.478583 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 16:52:28.478846 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.478826 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 16:52:28.521006 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.520981 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:28.587417 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.587384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/29dff70d-ffc3-4068-8d6d-c06ea66733ab-node-exporter-textfile\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.587548 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.587421 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6fmr\" (UniqueName: \"kubernetes.io/projected/29dff70d-ffc3-4068-8d6d-c06ea66733ab-kube-api-access-z6fmr\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.587548 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.587445 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29dff70d-ffc3-4068-8d6d-c06ea66733ab-sys\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.587548 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.587470 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/29dff70d-ffc3-4068-8d6d-c06ea66733ab-node-exporter-accelerators-collector-config\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.587548 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.587504 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29dff70d-ffc3-4068-8d6d-c06ea66733ab-metrics-client-ca\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.587548 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.587535 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/29dff70d-ffc3-4068-8d6d-c06ea66733ab-root\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.587733 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.587613 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29dff70d-ffc3-4068-8d6d-c06ea66733ab-node-exporter-tls\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.587733 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.587658 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/29dff70d-ffc3-4068-8d6d-c06ea66733ab-node-exporter-wtmp\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.587733 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.587687 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29dff70d-ffc3-4068-8d6d-c06ea66733ab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.645669 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.645644 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:28.688795 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.688772 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/29dff70d-ffc3-4068-8d6d-c06ea66733ab-root\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.688918 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.688813 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29dff70d-ffc3-4068-8d6d-c06ea66733ab-node-exporter-tls\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.688918 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.688835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/29dff70d-ffc3-4068-8d6d-c06ea66733ab-node-exporter-wtmp\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.688918 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.688863 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29dff70d-ffc3-4068-8d6d-c06ea66733ab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.688918 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.688891 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/29dff70d-ffc3-4068-8d6d-c06ea66733ab-node-exporter-textfile\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.688918 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.688896 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/29dff70d-ffc3-4068-8d6d-c06ea66733ab-root\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.689178 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.689003 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/29dff70d-ffc3-4068-8d6d-c06ea66733ab-node-exporter-wtmp\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.689178 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.688908 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6fmr\" (UniqueName: \"kubernetes.io/projected/29dff70d-ffc3-4068-8d6d-c06ea66733ab-kube-api-access-z6fmr\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.689178 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.689060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29dff70d-ffc3-4068-8d6d-c06ea66733ab-sys\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.689178 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.689110 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/29dff70d-ffc3-4068-8d6d-c06ea66733ab-node-exporter-accelerators-collector-config\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.689178 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.689150 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29dff70d-ffc3-4068-8d6d-c06ea66733ab-metrics-client-ca\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.689441 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.689223 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29dff70d-ffc3-4068-8d6d-c06ea66733ab-sys\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.689441 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.689276 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/29dff70d-ffc3-4068-8d6d-c06ea66733ab-node-exporter-textfile\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.689744 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.689723 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29dff70d-ffc3-4068-8d6d-c06ea66733ab-metrics-client-ca\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.689744 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.689736 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/29dff70d-ffc3-4068-8d6d-c06ea66733ab-node-exporter-accelerators-collector-config\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.691361 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.691339 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29dff70d-ffc3-4068-8d6d-c06ea66733ab-node-exporter-tls\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.691626 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.691610 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29dff70d-ffc3-4068-8d6d-c06ea66733ab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.704421 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.704393 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6fmr\" (UniqueName: \"kubernetes.io/projected/29dff70d-ffc3-4068-8d6d-c06ea66733ab-kube-api-access-z6fmr\") pod \"node-exporter-95brm\" (UID: \"29dff70d-ffc3-4068-8d6d-c06ea66733ab\") " pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.746458 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.746403 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:28.783957 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:28.783935 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-95brm" Jul 16 16:52:28.796020 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:52:28.795991 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29dff70d_ffc3_4068_8d6d_c06ea66733ab.slice/crio-093ac8e0e530c3212dfba23a970230138f5c590c6a7cc16a779136194e45b051 WatchSource:0}: Error finding container 093ac8e0e530c3212dfba23a970230138f5c590c6a7cc16a779136194e45b051: Status 404 returned error can't find the container with id 093ac8e0e530c3212dfba23a970230138f5c590c6a7cc16a779136194e45b051 Jul 16 16:52:29.017185 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:29.017111 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:29.109533 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:29.109510 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:29.350699 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:29.350613 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xql75" event={"ID":"e8c09583-4113-4f64-974f-03c3047db739","Type":"ContainerStarted","Data":"2f4b0e6f1bde1c442b938f1962f274a884497cbc23afae569ff9c6277906679c"} Jul 16 16:52:29.352021 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:29.351996 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-95brm" event={"ID":"29dff70d-ffc3-4068-8d6d-c06ea66733ab","Type":"ContainerStarted","Data":"093ac8e0e530c3212dfba23a970230138f5c590c6a7cc16a779136194e45b051"} Jul 16 16:52:29.371479 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:29.371430 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xql75" podStartSLOduration=36.627224181 podStartE2EDuration="48.371411294s" totalImagesPullingTime="11.744187113s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:51:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:29.370203486 +0000 UTC m=+86.888928716" watchObservedRunningTime="2026-07-16 16:52:29.371411294 +0000 UTC m=+86.890136538" Jul 16 16:52:29.466017 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:29.465981 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:29.567562 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:29.567525 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:29.686359 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:29.686324 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:29.928024 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:29.928001 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:30.014089 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:30.014030 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:30.246055 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:30.246026 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-644d85d8c9-qphqg" Jul 16 16:52:30.356633 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:30.356599 2575 generic.go:356] "Generic (PLEG): container finished" podID="29dff70d-ffc3-4068-8d6d-c06ea66733ab" containerID="5ce4335923879d166b7bc907e6c879514f0462e86bde020713ffbd41132bb4e2" exitCode=0 Jul 16 16:52:30.356992 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:30.356682 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-95brm" event={"ID":"29dff70d-ffc3-4068-8d6d-c06ea66733ab","Type":"ContainerDied","Data":"5ce4335923879d166b7bc907e6c879514f0462e86bde020713ffbd41132bb4e2"} Jul 16 16:52:30.431931 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:30.431903 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:30.527835 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:30.527806 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:31.132007 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:31.131974 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:31.233037 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:31.233006 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:31.362322 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:31.362285 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-95brm" event={"ID":"29dff70d-ffc3-4068-8d6d-c06ea66733ab","Type":"ContainerStarted","Data":"ccaa4b46c8d987730fe8426ccd6660210c117ee5b0d14e7521c19dbcb8670060"} Jul 16 16:52:31.362757 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:31.362343 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-95brm" event={"ID":"29dff70d-ffc3-4068-8d6d-c06ea66733ab","Type":"ContainerStarted","Data":"f037276804809b58af7fd1167ee826558be02f257bd8179791abe7d99e17ed35"} Jul 16 16:52:31.389754 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:31.389652 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-95brm" podStartSLOduration=2.334394139 podStartE2EDuration="3.389633625s" totalImagesPullingTime="992.49218ms" totalInitContainerRuntime="62.747306ms" isStatefulPod=true podCreationTimestamp="2026-07-16 16:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:31.387643457 +0000 UTC m=+88.906368702" watchObservedRunningTime="2026-07-16 16:52:31.389633625 +0000 UTC m=+88.908358858" Jul 16 16:52:31.437071 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:31.437033 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:31.531058 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:31.531022 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:31.633658 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:31.633631 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:31.735048 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:31.734982 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:32.390167 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:32.390138 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:32.499617 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:32.499584 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:33.203698 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.203671 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:33.235334 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.235297 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" podUID="c64d5338-cf37-4a5f-b064-6b090cd3cde8" containerName="registry" containerID="cri-o://3555ff3f7a5467c91b18900adea8b3cd45d3f2cff6b4137a05f85186553cbaad" gracePeriod=30 Jul 16 16:52:33.309536 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.309502 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:33.371737 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.371709 2575 generic.go:356] "Generic (PLEG): container finished" podID="c64d5338-cf37-4a5f-b064-6b090cd3cde8" containerID="3555ff3f7a5467c91b18900adea8b3cd45d3f2cff6b4137a05f85186553cbaad" exitCode=0 Jul 16 16:52:33.371856 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.371766 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" event={"ID":"c64d5338-cf37-4a5f-b064-6b090cd3cde8","Type":"ContainerDied","Data":"3555ff3f7a5467c91b18900adea8b3cd45d3f2cff6b4137a05f85186553cbaad"} Jul 16 16:52:33.475560 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.475536 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:52:33.630508 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.630476 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c64d5338-cf37-4a5f-b064-6b090cd3cde8-trusted-ca\") pod \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " Jul 16 16:52:33.630692 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.630524 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfp6g\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-kube-api-access-dfp6g\") pod \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " Jul 16 16:52:33.630692 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.630564 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls\") pod \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " Jul 16 16:52:33.630692 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.630587 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-certificates\") pod \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " Jul 16 16:52:33.630692 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.630605 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c64d5338-cf37-4a5f-b064-6b090cd3cde8-installation-pull-secrets\") pod \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " Jul 16 16:52:33.630692 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.630629 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c64d5338-cf37-4a5f-b064-6b090cd3cde8-image-registry-private-configuration\") pod \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " Jul 16 16:52:33.630692 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.630657 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-bound-sa-token\") pod \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " Jul 16 16:52:33.630692 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.630684 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c64d5338-cf37-4a5f-b064-6b090cd3cde8-ca-trust-extracted\") pod \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\" (UID: \"c64d5338-cf37-4a5f-b064-6b090cd3cde8\") " Jul 16 16:52:33.631123 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.631036 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c64d5338-cf37-4a5f-b064-6b090cd3cde8" (UID: "c64d5338-cf37-4a5f-b064-6b090cd3cde8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:52:33.631123 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.631083 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64d5338-cf37-4a5f-b064-6b090cd3cde8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c64d5338-cf37-4a5f-b064-6b090cd3cde8" (UID: "c64d5338-cf37-4a5f-b064-6b090cd3cde8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:52:33.633788 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.633754 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c64d5338-cf37-4a5f-b064-6b090cd3cde8" (UID: "c64d5338-cf37-4a5f-b064-6b090cd3cde8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:52:33.633788 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.633783 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64d5338-cf37-4a5f-b064-6b090cd3cde8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c64d5338-cf37-4a5f-b064-6b090cd3cde8" (UID: "c64d5338-cf37-4a5f-b064-6b090cd3cde8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:52:33.633967 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.633843 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-kube-api-access-dfp6g" (OuterVolumeSpecName: "kube-api-access-dfp6g") pod "c64d5338-cf37-4a5f-b064-6b090cd3cde8" (UID: "c64d5338-cf37-4a5f-b064-6b090cd3cde8"). InnerVolumeSpecName "kube-api-access-dfp6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:52:33.633967 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.633919 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64d5338-cf37-4a5f-b064-6b090cd3cde8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c64d5338-cf37-4a5f-b064-6b090cd3cde8" (UID: "c64d5338-cf37-4a5f-b064-6b090cd3cde8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:52:33.634051 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.633966 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c64d5338-cf37-4a5f-b064-6b090cd3cde8" (UID: "c64d5338-cf37-4a5f-b064-6b090cd3cde8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:52:33.642409 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.642384 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64d5338-cf37-4a5f-b064-6b090cd3cde8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c64d5338-cf37-4a5f-b064-6b090cd3cde8" (UID: "c64d5338-cf37-4a5f-b064-6b090cd3cde8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:52:33.732220 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.732150 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-tls\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:52:33.732220 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.732182 2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c64d5338-cf37-4a5f-b064-6b090cd3cde8-registry-certificates\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:52:33.732220 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.732199 2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c64d5338-cf37-4a5f-b064-6b090cd3cde8-installation-pull-secrets\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:52:33.732220 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.732214 2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c64d5338-cf37-4a5f-b064-6b090cd3cde8-image-registry-private-configuration\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:52:33.732415 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.732231 2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-bound-sa-token\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:52:33.732415 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.732244 2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c64d5338-cf37-4a5f-b064-6b090cd3cde8-ca-trust-extracted\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:52:33.732415 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.732257 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c64d5338-cf37-4a5f-b064-6b090cd3cde8-trusted-ca\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:52:33.732415 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.732270 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfp6g\" (UniqueName: \"kubernetes.io/projected/c64d5338-cf37-4a5f-b064-6b090cd3cde8-kube-api-access-dfp6g\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:52:33.768039 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:33.768010 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:34.380411 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:34.380381 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" Jul 16 16:52:34.380605 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:34.380380 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-789ddd76d-bbxxt" event={"ID":"c64d5338-cf37-4a5f-b064-6b090cd3cde8","Type":"ContainerDied","Data":"1e8a616b43a7edb17ecd4eed9670705957461b2a9954366e574c4cefdb85f5f2"} Jul 16 16:52:34.380605 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:34.380531 2575 scope.go:117] "RemoveContainer" containerID="3555ff3f7a5467c91b18900adea8b3cd45d3f2cff6b4137a05f85186553cbaad" Jul 16 16:52:34.397888 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:34.397869 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:34.403377 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:34.403356 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-789ddd76d-bbxxt"] Jul 16 16:52:34.407669 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:34.407651 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-789ddd76d-bbxxt"] Jul 16 16:52:34.747158 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:34.747050 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:34.907720 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:34.907686 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c64d5338-cf37-4a5f-b064-6b090cd3cde8" path="/var/lib/kubelet/pods/c64d5338-cf37-4a5f-b064-6b090cd3cde8/volumes" Jul 16 16:52:36.783923 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:36.783891 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:36.854794 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:36.854761 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:37.369990 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:37.369956 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:37.473192 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:37.473163 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:38.153849 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:38.153820 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:38.260689 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:38.260665 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:41.188482 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:41.188448 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:41.292110 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:41.292066 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:43.451985 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:43.451949 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:43.533741 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:43.533704 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:48.071046 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:48.071013 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:48.199809 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:48.199779 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:48.301956 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:48.301927 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:49.305317 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:49.305292 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2czkl" Jul 16 16:52:49.997819 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:49.997774 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:50.139945 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:50.139902 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:51.197485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:51.197431 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:51.272760 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:51.272733 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:52.005367 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:52.005334 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:52.099220 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:52.099188 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:52.827835 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:52.827809 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:55.941790 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:55.941749 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:56.018473 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:56.018443 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:56.468876 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:56.468845 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:52:56.553732 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:52:56.553702 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:53:01.553334 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:53:01.553300 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:53:01.630934 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:53:01.630907 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:53:10.992799 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:53:10.992769 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:53:11.096720 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:53:11.096690 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:53:47.774078 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:53:47.774040 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:53:49.431464 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:53:49.431435 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:53:56.701786 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:53:56.701703 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:53:56.800276 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:53:56.800239 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:53:56.871531 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:53:56.871492 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:53:57.440851 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:53:57.440820 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:53:57.536539 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:53:57.536504 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:54:07.882497 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:54:07.882445 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:54:08.010061 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:54:08.010031 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:54:27.214163 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:54:27.214124 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:54:27.324310 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:54:27.324279 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:55:03.125361 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:55:03.125323 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:55:03.199031 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:55:03.198998 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:55:05.787278 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:55:05.787245 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:55:05.886288 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:55:05.886260 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:55:15.024106 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:55:15.024076 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:55:15.105632 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:55:15.105602 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:55:17.137675 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:55:17.137646 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:55:17.239061 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:55:17.239033 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:55:17.315415 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:55:17.315382 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:55:25.948156 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:55:25.948059 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:55:26.910510 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:55:26.910476 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:56:02.815005 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:02.814974 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 16:56:02.815575 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:02.815383 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 16:56:02.822471 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:02.822449 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 16:56:15.376340 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:15.376313 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:56:18.263015 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:18.262981 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:56:26.835453 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:26.835420 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:56:37.781296 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:37.781262 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:56:38.104168 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:38.104139 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:56:41.755376 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:41.755341 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:56:41.858163 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:41.858116 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:56:43.631583 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:43.631540 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:56:43.738137 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:43.738067 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:56:44.087302 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:44.087220 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:56:44.225487 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:44.225456 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:56:46.636627 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:46.636587 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:56:46.865693 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:46.865666 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:56:49.541893 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:49.541866 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:56:49.618660 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:49.618629 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:56:53.831765 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:53.831728 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:56:55.042711 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:55.042632 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:56:55.136532 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:56:55.136500 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:01.334312 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:01.334278 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:01.426732 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:01.426704 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:03.434331 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:03.434298 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:15.596346 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:15.596315 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:15.695220 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:15.695192 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:16.176714 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.176683 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:16.381757 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.381723 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:16.485463 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.485399 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:16.528878 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.528848 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s"] Jul 16 16:57:16.529113 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.529082 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c64d5338-cf37-4a5f-b064-6b090cd3cde8" containerName="registry" Jul 16 16:57:16.529113 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.529108 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c64d5338-cf37-4a5f-b064-6b090cd3cde8" containerName="registry" Jul 16 16:57:16.529199 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.529154 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c64d5338-cf37-4a5f-b064-6b090cd3cde8" containerName="registry" Jul 16 16:57:16.531941 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.531909 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" Jul 16 16:57:16.534718 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.534697 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 16:57:16.534843 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.534808 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 16:57:16.534908 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.534882 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 16:57:16.534966 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.534926 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 16:57:16.535155 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.535142 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sdg7k\"" Jul 16 16:57:16.549443 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.549417 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s"] Jul 16 16:57:16.682877 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.682845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/83a24043-148b-46c0-977b-9011f90dbb66-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c7f465c76-x245s\" (UID: \"83a24043-148b-46c0-977b-9011f90dbb66\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" Jul 16 16:57:16.682877 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.682878 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83a24043-148b-46c0-977b-9011f90dbb66-tmp\") pod \"opendatahub-operator-controller-manager-c7f465c76-x245s\" (UID: \"83a24043-148b-46c0-977b-9011f90dbb66\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" Jul 16 16:57:16.683370 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.682949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83a24043-148b-46c0-977b-9011f90dbb66-webhook-cert\") pod \"opendatahub-operator-controller-manager-c7f465c76-x245s\" (UID: \"83a24043-148b-46c0-977b-9011f90dbb66\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" Jul 16 16:57:16.683370 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.682999 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/83a24043-148b-46c0-977b-9011f90dbb66-manifests\") pod \"opendatahub-operator-controller-manager-c7f465c76-x245s\" (UID: \"83a24043-148b-46c0-977b-9011f90dbb66\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" Jul 16 16:57:16.683370 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.683054 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-452x2\" (UniqueName: \"kubernetes.io/projected/83a24043-148b-46c0-977b-9011f90dbb66-kube-api-access-452x2\") pod \"opendatahub-operator-controller-manager-c7f465c76-x245s\" (UID: \"83a24043-148b-46c0-977b-9011f90dbb66\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" Jul 16 16:57:16.717868 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.717838 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:16.784257 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.784163 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83a24043-148b-46c0-977b-9011f90dbb66-webhook-cert\") pod \"opendatahub-operator-controller-manager-c7f465c76-x245s\" (UID: \"83a24043-148b-46c0-977b-9011f90dbb66\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" Jul 16 16:57:16.784257 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.784217 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/83a24043-148b-46c0-977b-9011f90dbb66-manifests\") pod \"opendatahub-operator-controller-manager-c7f465c76-x245s\" (UID: \"83a24043-148b-46c0-977b-9011f90dbb66\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" Jul 16 16:57:16.784452 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.784256 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-452x2\" (UniqueName: \"kubernetes.io/projected/83a24043-148b-46c0-977b-9011f90dbb66-kube-api-access-452x2\") pod \"opendatahub-operator-controller-manager-c7f465c76-x245s\" (UID: \"83a24043-148b-46c0-977b-9011f90dbb66\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" Jul 16 16:57:16.784452 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.784291 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/83a24043-148b-46c0-977b-9011f90dbb66-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c7f465c76-x245s\" (UID: \"83a24043-148b-46c0-977b-9011f90dbb66\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" Jul 16 16:57:16.784452 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.784314 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83a24043-148b-46c0-977b-9011f90dbb66-tmp\") pod \"opendatahub-operator-controller-manager-c7f465c76-x245s\" (UID: \"83a24043-148b-46c0-977b-9011f90dbb66\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" Jul 16 16:57:16.784672 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.784645 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/83a24043-148b-46c0-977b-9011f90dbb66-manifests\") pod \"opendatahub-operator-controller-manager-c7f465c76-x245s\" (UID: \"83a24043-148b-46c0-977b-9011f90dbb66\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" Jul 16 16:57:16.786617 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.786583 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83a24043-148b-46c0-977b-9011f90dbb66-tmp\") pod \"opendatahub-operator-controller-manager-c7f465c76-x245s\" (UID: \"83a24043-148b-46c0-977b-9011f90dbb66\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" Jul 16 16:57:16.786815 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.786796 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83a24043-148b-46c0-977b-9011f90dbb66-webhook-cert\") pod \"opendatahub-operator-controller-manager-c7f465c76-x245s\" (UID: \"83a24043-148b-46c0-977b-9011f90dbb66\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" Jul 16 16:57:16.786923 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.786903 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/83a24043-148b-46c0-977b-9011f90dbb66-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c7f465c76-x245s\" (UID: \"83a24043-148b-46c0-977b-9011f90dbb66\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" Jul 16 16:57:16.797304 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.797282 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-452x2\" (UniqueName: \"kubernetes.io/projected/83a24043-148b-46c0-977b-9011f90dbb66-kube-api-access-452x2\") pod \"opendatahub-operator-controller-manager-c7f465c76-x245s\" (UID: \"83a24043-148b-46c0-977b-9011f90dbb66\") " pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" Jul 16 16:57:16.815777 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.815753 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:16.841719 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.841698 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" Jul 16 16:57:16.917720 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.917697 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:16.981550 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.981529 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s"] Jul 16 16:57:16.983219 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:57:16.983194 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a24043_148b_46c0_977b_9011f90dbb66.slice/crio-f647fbf0b7d108072bfb9644db3c3ba72a647c09d16f5e6ba5fb0f7863c408be WatchSource:0}: Error finding container f647fbf0b7d108072bfb9644db3c3ba72a647c09d16f5e6ba5fb0f7863c408be: Status 404 returned error can't find the container with id f647fbf0b7d108072bfb9644db3c3ba72a647c09d16f5e6ba5fb0f7863c408be Jul 16 16:57:16.984913 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:16.984898 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 16:57:17.301685 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:17.301649 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" event={"ID":"83a24043-148b-46c0-977b-9011f90dbb66","Type":"ContainerStarted","Data":"f647fbf0b7d108072bfb9644db3c3ba72a647c09d16f5e6ba5fb0f7863c408be"} Jul 16 16:57:18.649649 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:18.649614 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:18.750127 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:18.750072 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:20.315191 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:20.315082 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" event={"ID":"83a24043-148b-46c0-977b-9011f90dbb66","Type":"ContainerStarted","Data":"de988057a177f9098743a09176c638eb98bf01d04873e044ac01da858b2d89a2"} Jul 16 16:57:21.320231 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:21.320140 2575 generic.go:356] "Generic (PLEG): container finished" podID="83a24043-148b-46c0-977b-9011f90dbb66" containerID="de988057a177f9098743a09176c638eb98bf01d04873e044ac01da858b2d89a2" exitCode=0 Jul 16 16:57:21.320231 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:21.320212 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" event={"ID":"83a24043-148b-46c0-977b-9011f90dbb66","Type":"ContainerDied","Data":"de988057a177f9098743a09176c638eb98bf01d04873e044ac01da858b2d89a2"} Jul 16 16:57:22.325981 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:22.325941 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" event={"ID":"83a24043-148b-46c0-977b-9011f90dbb66","Type":"ContainerStarted","Data":"a2680d84362ed7777257879c3584ace7e4b46d171a07c58119b8fec9d730182c"} Jul 16 16:57:22.326529 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:22.326112 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" Jul 16 16:57:22.352776 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:22.352713 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" podStartSLOduration=1.981933571 podStartE2EDuration="6.35269466s" totalImagesPullingTime="3.341821976s" totalInitContainerRuntime="1.028939113s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:57:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:57:22.350770362 +0000 UTC m=+379.869495593" watchObservedRunningTime="2026-07-16 16:57:22.35269466 +0000 UTC m=+379.871419893" Jul 16 16:57:23.321725 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:23.321694 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:23.420529 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:23.420470 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:32.076428 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:32.076368 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:32.179388 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:32.179352 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:33.331771 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:33.331739 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c7f465c76-x245s" Jul 16 16:57:35.731738 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:35.731710 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:35.846691 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:35.846657 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:36.017997 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.017922 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:36.155019 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.154993 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:36.158544 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.158516 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d8b857c67-wx2f8"] Jul 16 16:57:36.167108 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.167076 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d8b857c67-wx2f8" Jul 16 16:57:36.169925 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.169905 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 16 16:57:36.170955 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.170937 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 16:57:36.171057 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.170960 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-dv5l9\"" Jul 16 16:57:36.171057 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.170977 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 16:57:36.171592 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.171575 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 16 16:57:36.177988 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.177966 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d8b857c67-wx2f8"] Jul 16 16:57:36.233578 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.233544 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87dffba0-62da-4093-923b-ab29bfacdcd4-tmp\") pod \"kube-auth-proxy-5d8b857c67-wx2f8\" (UID: \"87dffba0-62da-4093-923b-ab29bfacdcd4\") " pod="openshift-ingress/kube-auth-proxy-5d8b857c67-wx2f8" Jul 16 16:57:36.233578 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.233579 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8jvm\" (UniqueName: \"kubernetes.io/projected/87dffba0-62da-4093-923b-ab29bfacdcd4-kube-api-access-t8jvm\") pod \"kube-auth-proxy-5d8b857c67-wx2f8\" (UID: \"87dffba0-62da-4093-923b-ab29bfacdcd4\") " pod="openshift-ingress/kube-auth-proxy-5d8b857c67-wx2f8" Jul 16 16:57:36.233774 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.233621 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87dffba0-62da-4093-923b-ab29bfacdcd4-tls-certs\") pod \"kube-auth-proxy-5d8b857c67-wx2f8\" (UID: \"87dffba0-62da-4093-923b-ab29bfacdcd4\") " pod="openshift-ingress/kube-auth-proxy-5d8b857c67-wx2f8" Jul 16 16:57:36.334814 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.334733 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87dffba0-62da-4093-923b-ab29bfacdcd4-tls-certs\") pod \"kube-auth-proxy-5d8b857c67-wx2f8\" (UID: \"87dffba0-62da-4093-923b-ab29bfacdcd4\") " pod="openshift-ingress/kube-auth-proxy-5d8b857c67-wx2f8" Jul 16 16:57:36.334814 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.334780 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87dffba0-62da-4093-923b-ab29bfacdcd4-tmp\") pod \"kube-auth-proxy-5d8b857c67-wx2f8\" (UID: \"87dffba0-62da-4093-923b-ab29bfacdcd4\") " pod="openshift-ingress/kube-auth-proxy-5d8b857c67-wx2f8" Jul 16 16:57:36.334814 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.334801 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8jvm\" (UniqueName: \"kubernetes.io/projected/87dffba0-62da-4093-923b-ab29bfacdcd4-kube-api-access-t8jvm\") pod \"kube-auth-proxy-5d8b857c67-wx2f8\" (UID: \"87dffba0-62da-4093-923b-ab29bfacdcd4\") " pod="openshift-ingress/kube-auth-proxy-5d8b857c67-wx2f8" Jul 16 16:57:36.337168 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.337134 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87dffba0-62da-4093-923b-ab29bfacdcd4-tmp\") pod \"kube-auth-proxy-5d8b857c67-wx2f8\" (UID: \"87dffba0-62da-4093-923b-ab29bfacdcd4\") " pod="openshift-ingress/kube-auth-proxy-5d8b857c67-wx2f8" Jul 16 16:57:36.337418 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.337399 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87dffba0-62da-4093-923b-ab29bfacdcd4-tls-certs\") pod \"kube-auth-proxy-5d8b857c67-wx2f8\" (UID: \"87dffba0-62da-4093-923b-ab29bfacdcd4\") " pod="openshift-ingress/kube-auth-proxy-5d8b857c67-wx2f8" Jul 16 16:57:36.354873 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.354839 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8jvm\" (UniqueName: \"kubernetes.io/projected/87dffba0-62da-4093-923b-ab29bfacdcd4-kube-api-access-t8jvm\") pod \"kube-auth-proxy-5d8b857c67-wx2f8\" (UID: \"87dffba0-62da-4093-923b-ab29bfacdcd4\") " pod="openshift-ingress/kube-auth-proxy-5d8b857c67-wx2f8" Jul 16 16:57:36.374921 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.374889 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-5mlts"] Jul 16 16:57:36.378251 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.378234 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-5mlts" Jul 16 16:57:36.380999 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.380980 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 16 16:57:36.381253 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.381239 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2pjnt\"" Jul 16 16:57:36.389263 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.389228 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-5mlts"] Jul 16 16:57:36.435274 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.435229 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/220953c4-02a2-4397-8e1a-53c65ac397ca-cert\") pod \"odh-model-controller-7998c67dc4-5mlts\" (UID: \"220953c4-02a2-4397-8e1a-53c65ac397ca\") " pod="opendatahub/odh-model-controller-7998c67dc4-5mlts" Jul 16 16:57:36.435445 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.435280 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kf5j\" (UniqueName: \"kubernetes.io/projected/220953c4-02a2-4397-8e1a-53c65ac397ca-kube-api-access-6kf5j\") pod \"odh-model-controller-7998c67dc4-5mlts\" (UID: \"220953c4-02a2-4397-8e1a-53c65ac397ca\") " pod="opendatahub/odh-model-controller-7998c67dc4-5mlts" Jul 16 16:57:36.440272 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.440244 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:36.477455 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.477210 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d8b857c67-wx2f8" Jul 16 16:57:36.533088 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.532950 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:36.536687 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.535913 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/220953c4-02a2-4397-8e1a-53c65ac397ca-cert\") pod \"odh-model-controller-7998c67dc4-5mlts\" (UID: \"220953c4-02a2-4397-8e1a-53c65ac397ca\") " pod="opendatahub/odh-model-controller-7998c67dc4-5mlts" Jul 16 16:57:36.536687 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.535964 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kf5j\" (UniqueName: \"kubernetes.io/projected/220953c4-02a2-4397-8e1a-53c65ac397ca-kube-api-access-6kf5j\") pod \"odh-model-controller-7998c67dc4-5mlts\" (UID: \"220953c4-02a2-4397-8e1a-53c65ac397ca\") " pod="opendatahub/odh-model-controller-7998c67dc4-5mlts" Jul 16 16:57:36.536687 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:57:36.536440 2575 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 16 16:57:36.536687 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:57:36.536506 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/220953c4-02a2-4397-8e1a-53c65ac397ca-cert podName:220953c4-02a2-4397-8e1a-53c65ac397ca nodeName:}" failed. No retries permitted until 2026-07-16 16:57:37.03648467 +0000 UTC m=+394.555209906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/220953c4-02a2-4397-8e1a-53c65ac397ca-cert") pod "odh-model-controller-7998c67dc4-5mlts" (UID: "220953c4-02a2-4397-8e1a-53c65ac397ca") : secret "odh-model-controller-webhook-cert" not found Jul 16 16:57:36.552413 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.552381 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kf5j\" (UniqueName: \"kubernetes.io/projected/220953c4-02a2-4397-8e1a-53c65ac397ca-kube-api-access-6kf5j\") pod \"odh-model-controller-7998c67dc4-5mlts\" (UID: \"220953c4-02a2-4397-8e1a-53c65ac397ca\") " pod="opendatahub/odh-model-controller-7998c67dc4-5mlts" Jul 16 16:57:36.629351 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.629318 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d8b857c67-wx2f8"] Jul 16 16:57:36.633184 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:57:36.633156 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87dffba0_62da_4093_923b_ab29bfacdcd4.slice/crio-6afc266e80a769f68afb233a717b10ac6d088d029ff53cd81f614e6ba8c9afda WatchSource:0}: Error finding container 6afc266e80a769f68afb233a717b10ac6d088d029ff53cd81f614e6ba8c9afda: Status 404 returned error can't find the container with id 6afc266e80a769f68afb233a717b10ac6d088d029ff53cd81f614e6ba8c9afda Jul 16 16:57:36.635982 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.635954 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:36.735135 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.735071 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:36.848075 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:36.848041 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:37.039669 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.039583 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/220953c4-02a2-4397-8e1a-53c65ac397ca-cert\") pod \"odh-model-controller-7998c67dc4-5mlts\" (UID: \"220953c4-02a2-4397-8e1a-53c65ac397ca\") " pod="opendatahub/odh-model-controller-7998c67dc4-5mlts" Jul 16 16:57:37.041941 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.041915 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/220953c4-02a2-4397-8e1a-53c65ac397ca-cert\") pod \"odh-model-controller-7998c67dc4-5mlts\" (UID: \"220953c4-02a2-4397-8e1a-53c65ac397ca\") " pod="opendatahub/odh-model-controller-7998c67dc4-5mlts" Jul 16 16:57:37.294327 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.294245 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-5mlts" Jul 16 16:57:37.379845 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.379667 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d8b857c67-wx2f8" event={"ID":"87dffba0-62da-4093-923b-ab29bfacdcd4","Type":"ContainerStarted","Data":"6afc266e80a769f68afb233a717b10ac6d088d029ff53cd81f614e6ba8c9afda"} Jul 16 16:57:37.420501 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.420462 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-d22dj"] Jul 16 16:57:37.423986 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.423965 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.425297 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.425274 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:37.426878 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.426740 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 16 16:57:37.426878 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.426751 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 16 16:57:37.427140 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.427118 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 16 16:57:37.427391 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.427374 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2dltg\"" Jul 16 16:57:37.427566 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.427550 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 16 16:57:37.438161 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.438130 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-d22dj"] Jul 16 16:57:37.443232 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.443209 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.443345 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.443262 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.443400 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.443338 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-token\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.443457 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.443411 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-cacerts\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.443457 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.443442 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhtzp\" (UniqueName: \"kubernetes.io/projected/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-kube-api-access-bhtzp\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.443545 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.443469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.443545 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.443520 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-local-certs\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.452912 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.452885 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-5mlts"] Jul 16 16:57:37.456175 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:57:37.456137 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220953c4_02a2_4397_8e1a_53c65ac397ca.slice/crio-6f1c0db1fc598d9b12743b37945632499cf2520ad300fc56c35110723dbacfcc WatchSource:0}: Error finding container 6f1c0db1fc598d9b12743b37945632499cf2520ad300fc56c35110723dbacfcc: Status 404 returned error can't find the container with id 6f1c0db1fc598d9b12743b37945632499cf2520ad300fc56c35110723dbacfcc Jul 16 16:57:37.537054 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.537005 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:37.544849 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.544741 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhtzp\" (UniqueName: \"kubernetes.io/projected/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-kube-api-access-bhtzp\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.545008 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.544843 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.545008 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.544908 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-local-certs\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.545008 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.544997 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.545208 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.545057 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.545208 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.545118 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-token\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.545208 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.545146 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-cacerts\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.546032 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.546006 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.548282 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.548257 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-cacerts\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.548642 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.548597 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.548753 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.548664 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.549157 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.549132 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-local-certs\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.560081 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.560025 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhtzp\" (UniqueName: \"kubernetes.io/projected/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-kube-api-access-bhtzp\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.563556 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.563489 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-token\") pod \"istiod-openshift-gateway-7574459656-d22dj\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.735534 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.735498 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:37.806408 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.806303 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:37.902018 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:57:37.901984 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50101ea9_f82b_4417_a0d8_b6d8b2e7419c.slice/crio-56c2376d60e494dff046aba522db7ae47834d4ac2e15c707c3ce6c771d3890ea WatchSource:0}: Error finding container 56c2376d60e494dff046aba522db7ae47834d4ac2e15c707c3ce6c771d3890ea: Status 404 returned error can't find the container with id 56c2376d60e494dff046aba522db7ae47834d4ac2e15c707c3ce6c771d3890ea Jul 16 16:57:37.906690 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:37.906666 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-d22dj"] Jul 16 16:57:38.387163 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:38.387126 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-5mlts" event={"ID":"220953c4-02a2-4397-8e1a-53c65ac397ca","Type":"ContainerStarted","Data":"6f1c0db1fc598d9b12743b37945632499cf2520ad300fc56c35110723dbacfcc"} Jul 16 16:57:38.390011 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:38.389950 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" event={"ID":"50101ea9-f82b-4417-a0d8-b6d8b2e7419c","Type":"ContainerStarted","Data":"56c2376d60e494dff046aba522db7ae47834d4ac2e15c707c3ce6c771d3890ea"} Jul 16 16:57:38.594573 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:38.594535 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:41.405340 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:41.405304 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d8b857c67-wx2f8" event={"ID":"87dffba0-62da-4093-923b-ab29bfacdcd4","Type":"ContainerStarted","Data":"987cc323438d97c6dfa2d4fd37b468afa00d44bbda6a8f6d8e7eb30c5b532232"} Jul 16 16:57:41.407158 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:41.407127 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-5mlts" event={"ID":"220953c4-02a2-4397-8e1a-53c65ac397ca","Type":"ContainerStarted","Data":"5e5f967f239d84b6e4fccd62fff015a45f18925b71702ba510696fd5fe50d9d9"} Jul 16 16:57:41.407302 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:41.407189 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-5mlts" Jul 16 16:57:41.448654 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:41.448515 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d8b857c67-wx2f8" podStartSLOduration=1.279673034 podStartE2EDuration="5.448493424s" totalImagesPullingTime="4.16882039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:57:41.42757704 +0000 UTC m=+398.946302279" watchObservedRunningTime="2026-07-16 16:57:41.448493424 +0000 UTC m=+398.967218660" Jul 16 16:57:41.448654 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:41.448642 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-5mlts" podStartSLOduration=2.047655369 podStartE2EDuration="5.448632612s" totalImagesPullingTime="3.400977243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:57:41.447922919 +0000 UTC m=+398.966648154" watchObservedRunningTime="2026-07-16 16:57:41.448632612 +0000 UTC m=+398.967357848" Jul 16 16:57:42.748862 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:42.748824 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:57:42.749255 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:42.748893 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:57:43.349312 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.349279 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:43.418462 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.418426 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" event={"ID":"50101ea9-f82b-4417-a0d8-b6d8b2e7419c","Type":"ContainerStarted","Data":"75c9d23703dc0b708929efeec7802a304a0d3c5386dbb71efe2d761e91bcc39c"} Jul 16 16:57:43.419187 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.419159 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:43.420908 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.420871 2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-d22dj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 16:57:43.421000 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.420933 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" podUID="50101ea9-f82b-4417-a0d8-b6d8b2e7419c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 16:57:43.443483 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.443428 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" podStartSLOduration=1.599152079 podStartE2EDuration="6.443409557s" totalImagesPullingTime="4.844257478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:57:43.441083531 +0000 UTC m=+400.959808790" watchObservedRunningTime="2026-07-16 16:57:43.443409557 +0000 UTC m=+400.962134790" Jul 16 16:57:43.447249 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.447221 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:43.678079 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.678046 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:43.758800 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.758765 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg"] Jul 16 16:57:43.762117 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.762071 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.764936 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.764916 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-vbhct\"" Jul 16 16:57:43.777803 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.777778 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg"] Jul 16 16:57:43.800503 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.800467 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc0f6a15-ceb2-40d1-a006-62c585469d63-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.800711 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.800545 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc0f6a15-ceb2-40d1-a006-62c585469d63-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.800711 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.800577 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc0f6a15-ceb2-40d1-a006-62c585469d63-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.800711 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.800609 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc0f6a15-ceb2-40d1-a006-62c585469d63-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.800711 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.800650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc0f6a15-ceb2-40d1-a006-62c585469d63-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.800929 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.800728 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc0f6a15-ceb2-40d1-a006-62c585469d63-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.800929 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.800764 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc0f6a15-ceb2-40d1-a006-62c585469d63-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.800929 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.800809 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc0f6a15-ceb2-40d1-a006-62c585469d63-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.800929 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.800860 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nxkg\" (UniqueName: \"kubernetes.io/projected/bc0f6a15-ceb2-40d1-a006-62c585469d63-kube-api-access-7nxkg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.827478 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.827448 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:43.901877 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.901845 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc0f6a15-ceb2-40d1-a006-62c585469d63-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.902057 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.901883 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc0f6a15-ceb2-40d1-a006-62c585469d63-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.902057 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.901920 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc0f6a15-ceb2-40d1-a006-62c585469d63-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.902057 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.901945 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc0f6a15-ceb2-40d1-a006-62c585469d63-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.902057 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.901985 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc0f6a15-ceb2-40d1-a006-62c585469d63-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.902332 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.902143 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nxkg\" (UniqueName: \"kubernetes.io/projected/bc0f6a15-ceb2-40d1-a006-62c585469d63-kube-api-access-7nxkg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.902332 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.902233 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc0f6a15-ceb2-40d1-a006-62c585469d63-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.902332 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.902284 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc0f6a15-ceb2-40d1-a006-62c585469d63-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.902332 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.902321 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc0f6a15-ceb2-40d1-a006-62c585469d63-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.902524 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.902348 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc0f6a15-ceb2-40d1-a006-62c585469d63-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.902524 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.902428 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc0f6a15-ceb2-40d1-a006-62c585469d63-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.902524 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.902460 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc0f6a15-ceb2-40d1-a006-62c585469d63-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.902685 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.902629 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc0f6a15-ceb2-40d1-a006-62c585469d63-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.902750 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.902733 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc0f6a15-ceb2-40d1-a006-62c585469d63-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.904720 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.904693 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc0f6a15-ceb2-40d1-a006-62c585469d63-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.904817 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.904803 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc0f6a15-ceb2-40d1-a006-62c585469d63-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.912221 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.912189 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nxkg\" (UniqueName: \"kubernetes.io/projected/bc0f6a15-ceb2-40d1-a006-62c585469d63-kube-api-access-7nxkg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.912336 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.912321 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc0f6a15-ceb2-40d1-a006-62c585469d63-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4qk2lg\" (UID: \"bc0f6a15-ceb2-40d1-a006-62c585469d63\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:43.932616 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:43.932545 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:44.072430 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:44.072392 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:44.241862 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:44.241832 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg"] Jul 16 16:57:44.244685 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:57:44.244660 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0f6a15_ceb2_40d1_a006_62c585469d63.slice/crio-5365b0d72a266744e77a81eb8284ac900396b4796d47448129955c222618bce1 WatchSource:0}: Error finding container 5365b0d72a266744e77a81eb8284ac900396b4796d47448129955c222618bce1: Status 404 returned error can't find the container with id 5365b0d72a266744e77a81eb8284ac900396b4796d47448129955c222618bce1 Jul 16 16:57:44.423459 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:44.423424 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" event={"ID":"bc0f6a15-ceb2-40d1-a006-62c585469d63","Type":"ContainerStarted","Data":"5365b0d72a266744e77a81eb8284ac900396b4796d47448129955c222618bce1"} Jul 16 16:57:44.424390 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:44.424368 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:46.910591 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:46.910560 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj"] Jul 16 16:57:46.921576 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:46.921542 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:46.922333 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:46.922276 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj"] Jul 16 16:57:47.031205 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.031166 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/713e019f-55eb-4f2c-b168-cde5bde99ecf-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.031371 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.031215 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/713e019f-55eb-4f2c-b168-cde5bde99ecf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.031371 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.031233 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713e019f-55eb-4f2c-b168-cde5bde99ecf-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.031371 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.031342 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/713e019f-55eb-4f2c-b168-cde5bde99ecf-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.031500 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.031389 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/713e019f-55eb-4f2c-b168-cde5bde99ecf-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.031500 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.031437 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/713e019f-55eb-4f2c-b168-cde5bde99ecf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.031500 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.031482 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkc8g\" (UniqueName: \"kubernetes.io/projected/713e019f-55eb-4f2c-b168-cde5bde99ecf-kube-api-access-bkc8g\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.035283 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.035251 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:57:47.035371 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.035334 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:57:47.035447 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.035382 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:57:47.132714 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.132670 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/713e019f-55eb-4f2c-b168-cde5bde99ecf-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.132897 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.132723 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/713e019f-55eb-4f2c-b168-cde5bde99ecf-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.132897 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.132767 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/713e019f-55eb-4f2c-b168-cde5bde99ecf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.132897 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.132816 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkc8g\" (UniqueName: \"kubernetes.io/projected/713e019f-55eb-4f2c-b168-cde5bde99ecf-kube-api-access-bkc8g\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.132897 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.132879 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/713e019f-55eb-4f2c-b168-cde5bde99ecf-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.133186 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.132928 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/713e019f-55eb-4f2c-b168-cde5bde99ecf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.133186 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.132962 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713e019f-55eb-4f2c-b168-cde5bde99ecf-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.134058 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.134002 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/713e019f-55eb-4f2c-b168-cde5bde99ecf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.135790 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.135762 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/713e019f-55eb-4f2c-b168-cde5bde99ecf-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.136150 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.136125 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/713e019f-55eb-4f2c-b168-cde5bde99ecf-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.136459 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.136437 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/713e019f-55eb-4f2c-b168-cde5bde99ecf-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.136861 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.136841 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/713e019f-55eb-4f2c-b168-cde5bde99ecf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.146144 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.145684 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/713e019f-55eb-4f2c-b168-cde5bde99ecf-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.150554 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.150502 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkc8g\" (UniqueName: \"kubernetes.io/projected/713e019f-55eb-4f2c-b168-cde5bde99ecf-kube-api-access-bkc8g\") pod \"istiod-openshift-gateway-94bb8fbfd-jnpsj\" (UID: \"713e019f-55eb-4f2c-b168-cde5bde99ecf\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.233787 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.233703 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:47.413580 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.413549 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj"] Jul 16 16:57:47.414459 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:57:47.414432 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713e019f_55eb_4f2c_b168_cde5bde99ecf.slice/crio-6e8f096bfced6aaa11ec0be79c489ba472b5c88f377f405859162ecf57a55621 WatchSource:0}: Error finding container 6e8f096bfced6aaa11ec0be79c489ba472b5c88f377f405859162ecf57a55621: Status 404 returned error can't find the container with id 6e8f096bfced6aaa11ec0be79c489ba472b5c88f377f405859162ecf57a55621 Jul 16 16:57:47.417452 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.417419 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:57:47.417532 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.417479 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:57:47.439773 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.439738 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" event={"ID":"bc0f6a15-ceb2-40d1-a006-62c585469d63","Type":"ContainerStarted","Data":"8cbbee81fd5bc261db2da52ffcea966f479dfdde009b2e2ad793e295b9e65a39"} Jul 16 16:57:47.441080 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.441057 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" event={"ID":"713e019f-55eb-4f2c-b168-cde5bde99ecf","Type":"ContainerStarted","Data":"6e8f096bfced6aaa11ec0be79c489ba472b5c88f377f405859162ecf57a55621"} Jul 16 16:57:47.505033 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:47.504910 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" podStartSLOduration=1.71655337 podStartE2EDuration="4.504886904s" totalImagesPullingTime="2.788333534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:57:47.497386626 +0000 UTC m=+405.016111859" watchObservedRunningTime="2026-07-16 16:57:47.504886904 +0000 UTC m=+405.023612140" Jul 16 16:57:48.072758 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:48.072715 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:48.077865 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:48.077840 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:48.446839 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:48.446798 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" event={"ID":"713e019f-55eb-4f2c-b168-cde5bde99ecf","Type":"ContainerStarted","Data":"663a8878971944b69ad860ad80050bcd8823b4e95c593c38597d8e1ed1ecb59e"} Jul 16 16:57:48.447132 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:48.447107 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:48.447955 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:48.447937 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4qk2lg" Jul 16 16:57:48.481940 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:48.481889 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" podStartSLOduration=2.481871284 podStartE2EDuration="2.481871284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:57:48.47880277 +0000 UTC m=+405.997528009" watchObservedRunningTime="2026-07-16 16:57:48.481871284 +0000 UTC m=+406.000596516" Jul 16 16:57:48.782858 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:48.782779 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:48.891103 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:48.891055 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:48.979231 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:48.979202 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:49.085280 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.085194 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:49.451521 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.451480 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:49.452961 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.452939 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jnpsj" Jul 16 16:57:49.554285 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.554255 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-d22dj"] Jul 16 16:57:49.554486 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.554465 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" podUID="50101ea9-f82b-4417-a0d8-b6d8b2e7419c" containerName="discovery" containerID="cri-o://75c9d23703dc0b708929efeec7802a304a0d3c5386dbb71efe2d761e91bcc39c" gracePeriod=30 Jul 16 16:57:49.794735 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.794712 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:49.860932 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.860901 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-local-certs\") pod \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " Jul 16 16:57:49.861077 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.860964 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-cacerts\") pod \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " Jul 16 16:57:49.861077 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.861012 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-token\") pod \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " Jul 16 16:57:49.861077 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.861051 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-kubeconfig\") pod \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " Jul 16 16:57:49.861239 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.861112 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-csr-ca-configmap\") pod \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " Jul 16 16:57:49.861239 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.861139 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhtzp\" (UniqueName: \"kubernetes.io/projected/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-kube-api-access-bhtzp\") pod \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " Jul 16 16:57:49.861239 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.861163 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-csr-dns-cert\") pod \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\" (UID: \"50101ea9-f82b-4417-a0d8-b6d8b2e7419c\") " Jul 16 16:57:49.861641 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.861614 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "50101ea9-f82b-4417-a0d8-b6d8b2e7419c" (UID: "50101ea9-f82b-4417-a0d8-b6d8b2e7419c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:57:49.863893 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.863844 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "50101ea9-f82b-4417-a0d8-b6d8b2e7419c" (UID: "50101ea9-f82b-4417-a0d8-b6d8b2e7419c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:57:49.863893 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.863850 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-local-certs" (OuterVolumeSpecName: "local-certs") pod "50101ea9-f82b-4417-a0d8-b6d8b2e7419c" (UID: "50101ea9-f82b-4417-a0d8-b6d8b2e7419c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:57:49.864015 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.863900 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-kube-api-access-bhtzp" (OuterVolumeSpecName: "kube-api-access-bhtzp") pod "50101ea9-f82b-4417-a0d8-b6d8b2e7419c" (UID: "50101ea9-f82b-4417-a0d8-b6d8b2e7419c"). InnerVolumeSpecName "kube-api-access-bhtzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:57:49.864015 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.863910 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-token" (OuterVolumeSpecName: "istio-token") pod "50101ea9-f82b-4417-a0d8-b6d8b2e7419c" (UID: "50101ea9-f82b-4417-a0d8-b6d8b2e7419c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:57:49.864015 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.863915 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "50101ea9-f82b-4417-a0d8-b6d8b2e7419c" (UID: "50101ea9-f82b-4417-a0d8-b6d8b2e7419c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:57:49.864015 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.863936 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-cacerts" (OuterVolumeSpecName: "cacerts") pod "50101ea9-f82b-4417-a0d8-b6d8b2e7419c" (UID: "50101ea9-f82b-4417-a0d8-b6d8b2e7419c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:57:49.962707 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.962666 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-csr-ca-configmap\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:57:49.962902 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.962776 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhtzp\" (UniqueName: \"kubernetes.io/projected/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-kube-api-access-bhtzp\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:57:49.962902 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.962796 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-csr-dns-cert\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:57:49.962902 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.962806 2575 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-local-certs\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:57:49.962902 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.962814 2575 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-cacerts\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:57:49.962902 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.962822 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-token\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:57:49.962902 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:49.962832 2575 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50101ea9-f82b-4417-a0d8-b6d8b2e7419c-istio-kubeconfig\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:57:50.135016 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:50.134982 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:50.239767 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:50.239726 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:50.338686 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:50.338655 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:57:50.456676 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:50.456594 2575 generic.go:356] "Generic (PLEG): container finished" podID="50101ea9-f82b-4417-a0d8-b6d8b2e7419c" containerID="75c9d23703dc0b708929efeec7802a304a0d3c5386dbb71efe2d761e91bcc39c" exitCode=0 Jul 16 16:57:50.456676 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:50.456637 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" Jul 16 16:57:50.456848 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:50.456683 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" event={"ID":"50101ea9-f82b-4417-a0d8-b6d8b2e7419c","Type":"ContainerDied","Data":"75c9d23703dc0b708929efeec7802a304a0d3c5386dbb71efe2d761e91bcc39c"} Jul 16 16:57:50.456848 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:50.456733 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-d22dj" event={"ID":"50101ea9-f82b-4417-a0d8-b6d8b2e7419c","Type":"ContainerDied","Data":"56c2376d60e494dff046aba522db7ae47834d4ac2e15c707c3ce6c771d3890ea"} Jul 16 16:57:50.456848 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:50.456762 2575 scope.go:117] "RemoveContainer" containerID="75c9d23703dc0b708929efeec7802a304a0d3c5386dbb71efe2d761e91bcc39c" Jul 16 16:57:50.464950 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:50.464923 2575 scope.go:117] "RemoveContainer" containerID="75c9d23703dc0b708929efeec7802a304a0d3c5386dbb71efe2d761e91bcc39c" Jul 16 16:57:50.465250 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:57:50.465232 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c9d23703dc0b708929efeec7802a304a0d3c5386dbb71efe2d761e91bcc39c\": container with ID starting with 75c9d23703dc0b708929efeec7802a304a0d3c5386dbb71efe2d761e91bcc39c not found: ID does not exist" containerID="75c9d23703dc0b708929efeec7802a304a0d3c5386dbb71efe2d761e91bcc39c" Jul 16 16:57:50.465307 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:50.465258 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c9d23703dc0b708929efeec7802a304a0d3c5386dbb71efe2d761e91bcc39c"} err="failed to get container status \"75c9d23703dc0b708929efeec7802a304a0d3c5386dbb71efe2d761e91bcc39c\": rpc error: code = NotFound desc = could not find container \"75c9d23703dc0b708929efeec7802a304a0d3c5386dbb71efe2d761e91bcc39c\": container with ID starting with 75c9d23703dc0b708929efeec7802a304a0d3c5386dbb71efe2d761e91bcc39c not found: ID does not exist" Jul 16 16:57:50.484710 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:50.484683 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-d22dj"] Jul 16 16:57:50.491257 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:50.491233 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-d22dj"] Jul 16 16:57:50.908782 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:50.908750 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50101ea9-f82b-4417-a0d8-b6d8b2e7419c" path="/var/lib/kubelet/pods/50101ea9-f82b-4417-a0d8-b6d8b2e7419c/volumes" Jul 16 16:57:52.413393 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:57:52.413367 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-5mlts" Jul 16 16:58:30.067476 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:30.067396 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:58:30.137901 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:30.137867 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:58:37.207960 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.207930 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:58:37.293040 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.293004 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:58:37.357538 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.357502 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zh2wk"] Jul 16 16:58:37.357844 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.357828 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50101ea9-f82b-4417-a0d8-b6d8b2e7419c" containerName="discovery" Jul 16 16:58:37.357917 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.357847 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="50101ea9-f82b-4417-a0d8-b6d8b2e7419c" containerName="discovery" Jul 16 16:58:37.357967 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.357936 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="50101ea9-f82b-4417-a0d8-b6d8b2e7419c" containerName="discovery" Jul 16 16:58:37.360943 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.360921 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zh2wk" Jul 16 16:58:37.365296 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.365269 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-6gzgl\"" Jul 16 16:58:37.366371 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.366350 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 16:58:37.366477 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.366357 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 16:58:37.375041 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.375014 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zh2wk"] Jul 16 16:58:37.411960 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.411930 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:58:37.460394 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.460308 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4jv9\" (UniqueName: \"kubernetes.io/projected/d6f150ad-3296-416c-9288-942b7a022f69-kube-api-access-c4jv9\") pod \"kuadrant-operator-catalog-zh2wk\" (UID: \"d6f150ad-3296-416c-9288-942b7a022f69\") " pod="kuadrant-system/kuadrant-operator-catalog-zh2wk" Jul 16 16:58:37.561414 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.561383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4jv9\" (UniqueName: \"kubernetes.io/projected/d6f150ad-3296-416c-9288-942b7a022f69-kube-api-access-c4jv9\") pod \"kuadrant-operator-catalog-zh2wk\" (UID: \"d6f150ad-3296-416c-9288-942b7a022f69\") " pod="kuadrant-system/kuadrant-operator-catalog-zh2wk" Jul 16 16:58:37.571736 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.571705 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4jv9\" (UniqueName: \"kubernetes.io/projected/d6f150ad-3296-416c-9288-942b7a022f69-kube-api-access-c4jv9\") pod \"kuadrant-operator-catalog-zh2wk\" (UID: \"d6f150ad-3296-416c-9288-942b7a022f69\") " pod="kuadrant-system/kuadrant-operator-catalog-zh2wk" Jul 16 16:58:37.671317 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.671285 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zh2wk" Jul 16 16:58:37.698809 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.698779 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zh2wk"] Jul 16 16:58:37.800273 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.800252 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zh2wk"] Jul 16 16:58:37.802509 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:58:37.802482 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f150ad_3296_416c_9288_942b7a022f69.slice/crio-a25d9de19269e2bc8fa9d7da47479527b81fb55d66b93f705a41e6894e94ef2a WatchSource:0}: Error finding container a25d9de19269e2bc8fa9d7da47479527b81fb55d66b93f705a41e6894e94ef2a: Status 404 returned error can't find the container with id a25d9de19269e2bc8fa9d7da47479527b81fb55d66b93f705a41e6894e94ef2a Jul 16 16:58:37.911824 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.911789 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vlxlv"] Jul 16 16:58:37.916032 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.916016 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vlxlv" Jul 16 16:58:37.924255 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.924231 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vlxlv"] Jul 16 16:58:37.964256 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:37.964231 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nwvj\" (UniqueName: \"kubernetes.io/projected/12c5a5d4-0d98-4acf-9809-267521565c51-kube-api-access-7nwvj\") pod \"kuadrant-operator-catalog-vlxlv\" (UID: \"12c5a5d4-0d98-4acf-9809-267521565c51\") " pod="kuadrant-system/kuadrant-operator-catalog-vlxlv" Jul 16 16:58:38.065284 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:38.065196 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nwvj\" (UniqueName: \"kubernetes.io/projected/12c5a5d4-0d98-4acf-9809-267521565c51-kube-api-access-7nwvj\") pod \"kuadrant-operator-catalog-vlxlv\" (UID: \"12c5a5d4-0d98-4acf-9809-267521565c51\") " pod="kuadrant-system/kuadrant-operator-catalog-vlxlv" Jul 16 16:58:38.075725 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:38.075695 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nwvj\" (UniqueName: \"kubernetes.io/projected/12c5a5d4-0d98-4acf-9809-267521565c51-kube-api-access-7nwvj\") pod \"kuadrant-operator-catalog-vlxlv\" (UID: \"12c5a5d4-0d98-4acf-9809-267521565c51\") " pod="kuadrant-system/kuadrant-operator-catalog-vlxlv" Jul 16 16:58:38.225982 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:38.225950 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vlxlv" Jul 16 16:58:38.391265 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:38.391235 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vlxlv"] Jul 16 16:58:38.472517 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:58:38.472487 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c5a5d4_0d98_4acf_9809_267521565c51.slice/crio-da277fe9127da7edbd3416c4f1df983031d50658d4558f8f76242e0b541c688e WatchSource:0}: Error finding container da277fe9127da7edbd3416c4f1df983031d50658d4558f8f76242e0b541c688e: Status 404 returned error can't find the container with id da277fe9127da7edbd3416c4f1df983031d50658d4558f8f76242e0b541c688e Jul 16 16:58:38.647890 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:38.647856 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vlxlv" event={"ID":"12c5a5d4-0d98-4acf-9809-267521565c51","Type":"ContainerStarted","Data":"da277fe9127da7edbd3416c4f1df983031d50658d4558f8f76242e0b541c688e"} Jul 16 16:58:38.648993 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:38.648968 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zh2wk" event={"ID":"d6f150ad-3296-416c-9288-942b7a022f69","Type":"ContainerStarted","Data":"a25d9de19269e2bc8fa9d7da47479527b81fb55d66b93f705a41e6894e94ef2a"} Jul 16 16:58:40.657472 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:40.657435 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zh2wk" event={"ID":"d6f150ad-3296-416c-9288-942b7a022f69","Type":"ContainerStarted","Data":"e69a4e3713ff5daf506d91e628f320a140c8b17245d08dc4f780c8c53ee46e64"} Jul 16 16:58:40.657873 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:40.657530 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-zh2wk" podUID="d6f150ad-3296-416c-9288-942b7a022f69" containerName="registry-server" containerID="cri-o://e69a4e3713ff5daf506d91e628f320a140c8b17245d08dc4f780c8c53ee46e64" gracePeriod=2 Jul 16 16:58:40.676880 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:40.676833 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-zh2wk" podStartSLOduration=1.750958941 podStartE2EDuration="3.676815712s" totalImagesPullingTime="1.925856771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:58:40.67628343 +0000 UTC m=+458.195008664" watchObservedRunningTime="2026-07-16 16:58:40.676815712 +0000 UTC m=+458.195540946" Jul 16 16:58:40.896406 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:40.896384 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zh2wk" Jul 16 16:58:40.989366 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:40.989271 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4jv9\" (UniqueName: \"kubernetes.io/projected/d6f150ad-3296-416c-9288-942b7a022f69-kube-api-access-c4jv9\") pod \"d6f150ad-3296-416c-9288-942b7a022f69\" (UID: \"d6f150ad-3296-416c-9288-942b7a022f69\") " Jul 16 16:58:40.991644 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:40.991619 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f150ad-3296-416c-9288-942b7a022f69-kube-api-access-c4jv9" (OuterVolumeSpecName: "kube-api-access-c4jv9") pod "d6f150ad-3296-416c-9288-942b7a022f69" (UID: "d6f150ad-3296-416c-9288-942b7a022f69"). InnerVolumeSpecName "kube-api-access-c4jv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:58:41.090999 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:41.090959 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4jv9\" (UniqueName: \"kubernetes.io/projected/d6f150ad-3296-416c-9288-942b7a022f69-kube-api-access-c4jv9\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:58:41.662334 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:41.662301 2575 generic.go:356] "Generic (PLEG): container finished" podID="d6f150ad-3296-416c-9288-942b7a022f69" containerID="e69a4e3713ff5daf506d91e628f320a140c8b17245d08dc4f780c8c53ee46e64" exitCode=0 Jul 16 16:58:41.662792 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:41.662355 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zh2wk" Jul 16 16:58:41.662792 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:41.662386 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zh2wk" event={"ID":"d6f150ad-3296-416c-9288-942b7a022f69","Type":"ContainerDied","Data":"e69a4e3713ff5daf506d91e628f320a140c8b17245d08dc4f780c8c53ee46e64"} Jul 16 16:58:41.662792 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:41.662438 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zh2wk" event={"ID":"d6f150ad-3296-416c-9288-942b7a022f69","Type":"ContainerDied","Data":"a25d9de19269e2bc8fa9d7da47479527b81fb55d66b93f705a41e6894e94ef2a"} Jul 16 16:58:41.662792 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:41.662457 2575 scope.go:117] "RemoveContainer" containerID="e69a4e3713ff5daf506d91e628f320a140c8b17245d08dc4f780c8c53ee46e64" Jul 16 16:58:41.670601 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:41.670586 2575 scope.go:117] "RemoveContainer" containerID="e69a4e3713ff5daf506d91e628f320a140c8b17245d08dc4f780c8c53ee46e64" Jul 16 16:58:41.670825 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:58:41.670808 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69a4e3713ff5daf506d91e628f320a140c8b17245d08dc4f780c8c53ee46e64\": container with ID starting with e69a4e3713ff5daf506d91e628f320a140c8b17245d08dc4f780c8c53ee46e64 not found: ID does not exist" containerID="e69a4e3713ff5daf506d91e628f320a140c8b17245d08dc4f780c8c53ee46e64" Jul 16 16:58:41.670876 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:41.670834 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69a4e3713ff5daf506d91e628f320a140c8b17245d08dc4f780c8c53ee46e64"} err="failed to get container status \"e69a4e3713ff5daf506d91e628f320a140c8b17245d08dc4f780c8c53ee46e64\": rpc error: code = NotFound desc = could not find container \"e69a4e3713ff5daf506d91e628f320a140c8b17245d08dc4f780c8c53ee46e64\": container with ID starting with e69a4e3713ff5daf506d91e628f320a140c8b17245d08dc4f780c8c53ee46e64 not found: ID does not exist" Jul 16 16:58:41.683991 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:41.683971 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zh2wk"] Jul 16 16:58:41.688444 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:41.688426 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zh2wk"] Jul 16 16:58:42.667814 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:42.667723 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vlxlv" event={"ID":"12c5a5d4-0d98-4acf-9809-267521565c51","Type":"ContainerStarted","Data":"f9da37454729bb4780dc30d90ae2ba36b1e488cb32942055716f451ff6327890"} Jul 16 16:58:42.688885 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:42.688836 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-vlxlv" podStartSLOduration=1.775670257 podStartE2EDuration="5.688819106s" totalImagesPullingTime="3.913148849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:58:42.687337865 +0000 UTC m=+460.206063100" watchObservedRunningTime="2026-07-16 16:58:42.688819106 +0000 UTC m=+460.207544341" Jul 16 16:58:42.908749 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:42.908718 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6f150ad-3296-416c-9288-942b7a022f69" path="/var/lib/kubelet/pods/d6f150ad-3296-416c-9288-942b7a022f69/volumes" Jul 16 16:58:48.226955 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:48.226923 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-vlxlv" Jul 16 16:58:48.227450 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:48.227145 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-vlxlv" Jul 16 16:58:48.249757 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:48.249731 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-vlxlv" Jul 16 16:58:48.712196 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:58:48.712165 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-vlxlv" Jul 16 16:59:02.429520 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:02.429486 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:06.491769 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:06.491735 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:09.135968 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:09.135928 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pq64f"] Jul 16 16:59:09.136463 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:09.136226 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6f150ad-3296-416c-9288-942b7a022f69" containerName="registry-server" Jul 16 16:59:09.136463 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:09.136237 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6f150ad-3296-416c-9288-942b7a022f69" containerName="registry-server" Jul 16 16:59:09.136463 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:09.136311 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6f150ad-3296-416c-9288-942b7a022f69" containerName="registry-server" Jul 16 16:59:09.142402 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:09.142384 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pq64f" Jul 16 16:59:09.153668 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:09.153647 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-648nk\"" Jul 16 16:59:09.154360 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:09.154343 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 16 16:59:09.157030 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:09.157009 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pq64f"] Jul 16 16:59:09.227373 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:09.227334 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhr29\" (UniqueName: \"kubernetes.io/projected/0a789618-3fda-44e6-a1a1-29ad597510cc-kube-api-access-fhr29\") pod \"dns-operator-controller-manager-864dd78c4c-pq64f\" (UID: \"0a789618-3fda-44e6-a1a1-29ad597510cc\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pq64f" Jul 16 16:59:09.327998 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:09.327968 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhr29\" (UniqueName: \"kubernetes.io/projected/0a789618-3fda-44e6-a1a1-29ad597510cc-kube-api-access-fhr29\") pod \"dns-operator-controller-manager-864dd78c4c-pq64f\" (UID: \"0a789618-3fda-44e6-a1a1-29ad597510cc\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pq64f" Jul 16 16:59:09.340270 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:09.340238 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhr29\" (UniqueName: \"kubernetes.io/projected/0a789618-3fda-44e6-a1a1-29ad597510cc-kube-api-access-fhr29\") pod \"dns-operator-controller-manager-864dd78c4c-pq64f\" (UID: \"0a789618-3fda-44e6-a1a1-29ad597510cc\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pq64f" Jul 16 16:59:09.451860 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:09.451769 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pq64f" Jul 16 16:59:09.587204 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:09.587183 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pq64f"] Jul 16 16:59:09.589859 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:59:09.589834 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a789618_3fda_44e6_a1a1_29ad597510cc.slice/crio-bd5a2e04f342d523b0fccb6065f64bb81cfddaf8557f48526831179d5d0fd750 WatchSource:0}: Error finding container bd5a2e04f342d523b0fccb6065f64bb81cfddaf8557f48526831179d5d0fd750: Status 404 returned error can't find the container with id bd5a2e04f342d523b0fccb6065f64bb81cfddaf8557f48526831179d5d0fd750 Jul 16 16:59:09.774208 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:09.774119 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pq64f" event={"ID":"0a789618-3fda-44e6-a1a1-29ad597510cc","Type":"ContainerStarted","Data":"bd5a2e04f342d523b0fccb6065f64bb81cfddaf8557f48526831179d5d0fd750"} Jul 16 16:59:10.021632 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:10.021601 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:12.791373 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:12.791336 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pq64f" event={"ID":"0a789618-3fda-44e6-a1a1-29ad597510cc","Type":"ContainerStarted","Data":"b6e81eee2f04b5e59895783d20c78f4a3edc06d971480d25af1bd76e906ab884"} Jul 16 16:59:12.791755 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:12.791469 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pq64f" Jul 16 16:59:12.817559 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:12.817507 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pq64f" podStartSLOduration=1.559603528 podStartE2EDuration="3.817489717s" totalImagesPullingTime="2.257886189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:59:12.816318085 +0000 UTC m=+490.335043320" watchObservedRunningTime="2026-07-16 16:59:12.817489717 +0000 UTC m=+490.336214992" Jul 16 16:59:13.058171 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:13.058055 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:13.283911 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:13.283868 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:13.389762 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:13.389732 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:13.497519 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:13.497486 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:14.531088 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:14.531041 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:14.559413 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:14.559385 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct"] Jul 16 16:59:14.565817 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:14.565797 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:14.570330 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:14.570307 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tjtrx\"" Jul 16 16:59:14.583323 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:14.583292 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct"] Jul 16 16:59:14.675328 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:14.675289 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpzxt\" (UniqueName: \"kubernetes.io/projected/4885f2d3-f2af-4c60-b452-e1fa9d853d69-kube-api-access-zpzxt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-pbxct\" (UID: \"4885f2d3-f2af-4c60-b452-e1fa9d853d69\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:14.675328 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:14.675334 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4885f2d3-f2af-4c60-b452-e1fa9d853d69-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-pbxct\" (UID: \"4885f2d3-f2af-4c60-b452-e1fa9d853d69\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:14.776149 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:14.776109 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpzxt\" (UniqueName: \"kubernetes.io/projected/4885f2d3-f2af-4c60-b452-e1fa9d853d69-kube-api-access-zpzxt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-pbxct\" (UID: \"4885f2d3-f2af-4c60-b452-e1fa9d853d69\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:14.776312 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:14.776169 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4885f2d3-f2af-4c60-b452-e1fa9d853d69-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-pbxct\" (UID: \"4885f2d3-f2af-4c60-b452-e1fa9d853d69\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:14.776595 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:14.776572 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4885f2d3-f2af-4c60-b452-e1fa9d853d69-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-pbxct\" (UID: \"4885f2d3-f2af-4c60-b452-e1fa9d853d69\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:14.785669 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:14.785611 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpzxt\" (UniqueName: \"kubernetes.io/projected/4885f2d3-f2af-4c60-b452-e1fa9d853d69-kube-api-access-zpzxt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-pbxct\" (UID: \"4885f2d3-f2af-4c60-b452-e1fa9d853d69\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:14.875439 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:14.875399 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:14.969485 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:14.969391 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:15.010769 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:15.010741 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct"] Jul 16 16:59:15.014306 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:59:15.014277 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4885f2d3_f2af_4c60_b452_e1fa9d853d69.slice/crio-f2866cef3af9b1244271056bb20c5b0b8118c95fd11e224c40de8cfd29bea772 WatchSource:0}: Error finding container f2866cef3af9b1244271056bb20c5b0b8118c95fd11e224c40de8cfd29bea772: Status 404 returned error can't find the container with id f2866cef3af9b1244271056bb20c5b0b8118c95fd11e224c40de8cfd29bea772 Jul 16 16:59:15.459719 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:15.459661 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:15.805788 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:15.805704 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" event={"ID":"4885f2d3-f2af-4c60-b452-e1fa9d853d69","Type":"ContainerStarted","Data":"f2866cef3af9b1244271056bb20c5b0b8118c95fd11e224c40de8cfd29bea772"} Jul 16 16:59:18.017086 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:18.017054 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:18.168731 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:18.168698 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:18.635435 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:18.635376 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:20.136085 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:20.136043 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:20.414507 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:20.414409 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:20.646864 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:20.646827 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh"] Jul 16 16:59:20.650625 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:20.650597 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" Jul 16 16:59:20.654519 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:20.654484 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qz2zp\"" Jul 16 16:59:20.664287 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:20.664259 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh"] Jul 16 16:59:20.679381 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:20.679317 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:20.831135 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:20.831087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk4n6\" (UniqueName: \"kubernetes.io/projected/c0929e19-dae5-452a-a636-4c61bdec2a8d-kube-api-access-jk4n6\") pod \"limitador-operator-controller-manager-859d7b78bd-n2prh\" (UID: \"c0929e19-dae5-452a-a636-4c61bdec2a8d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" Jul 16 16:59:20.931694 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:20.931602 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk4n6\" (UniqueName: \"kubernetes.io/projected/c0929e19-dae5-452a-a636-4c61bdec2a8d-kube-api-access-jk4n6\") pod \"limitador-operator-controller-manager-859d7b78bd-n2prh\" (UID: \"c0929e19-dae5-452a-a636-4c61bdec2a8d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" Jul 16 16:59:20.954204 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:20.954164 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk4n6\" (UniqueName: \"kubernetes.io/projected/c0929e19-dae5-452a-a636-4c61bdec2a8d-kube-api-access-jk4n6\") pod \"limitador-operator-controller-manager-859d7b78bd-n2prh\" (UID: \"c0929e19-dae5-452a-a636-4c61bdec2a8d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" Jul 16 16:59:20.963204 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:20.963176 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" Jul 16 16:59:21.421226 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:21.421200 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh"] Jul 16 16:59:21.424309 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:59:21.424283 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0929e19_dae5_452a_a636_4c61bdec2a8d.slice/crio-f07629574f82bb6090d9b42a01df6e3e35f15357c87936bcc0adf2518c22cd02 WatchSource:0}: Error finding container f07629574f82bb6090d9b42a01df6e3e35f15357c87936bcc0adf2518c22cd02: Status 404 returned error can't find the container with id f07629574f82bb6090d9b42a01df6e3e35f15357c87936bcc0adf2518c22cd02 Jul 16 16:59:21.834197 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:21.834155 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" event={"ID":"4885f2d3-f2af-4c60-b452-e1fa9d853d69","Type":"ContainerStarted","Data":"5116c63b3c8258f1e946e88debb46a99fa525d492d9555d45fdc1c4d3729b0c4"} Jul 16 16:59:21.834385 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:21.834272 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:21.835448 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:21.835427 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" event={"ID":"c0929e19-dae5-452a-a636-4c61bdec2a8d","Type":"ContainerStarted","Data":"f07629574f82bb6090d9b42a01df6e3e35f15357c87936bcc0adf2518c22cd02"} Jul 16 16:59:21.859782 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:21.859735 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" podStartSLOduration=1.5298547660000001 podStartE2EDuration="7.859720014s" totalImagesPullingTime="6.329865248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:59:21.857225412 +0000 UTC m=+499.375950646" watchObservedRunningTime="2026-07-16 16:59:21.859720014 +0000 UTC m=+499.378445248" Jul 16 16:59:22.191762 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:22.191728 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:22.336208 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:22.336170 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:23.797689 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:23.797654 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pq64f" Jul 16 16:59:23.848172 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:23.848135 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" event={"ID":"c0929e19-dae5-452a-a636-4c61bdec2a8d","Type":"ContainerStarted","Data":"8a963ca0c131cc23102f115f319078ed2ea877424bb8a7781f7e0829edc8ed61"} Jul 16 16:59:23.848347 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:23.848259 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" Jul 16 16:59:23.875024 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:23.874952 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" podStartSLOduration=2.306914039 podStartE2EDuration="3.874932377s" totalImagesPullingTime="1.568018338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:59:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:59:23.874345835 +0000 UTC m=+501.393071061" watchObservedRunningTime="2026-07-16 16:59:23.874932377 +0000 UTC m=+501.393657612" Jul 16 16:59:24.501652 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:24.501615 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:24.619554 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:24.619525 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:26.036681 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:26.036646 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:26.133862 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:26.133827 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:32.842560 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:32.842531 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:34.854370 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:34.854334 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" Jul 16 16:59:35.168389 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.168306 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct"] Jul 16 16:59:35.168544 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.168508 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" podUID="4885f2d3-f2af-4c60-b452-e1fa9d853d69" containerName="manager" containerID="cri-o://5116c63b3c8258f1e946e88debb46a99fa525d492d9555d45fdc1c4d3729b0c4" gracePeriod=2 Jul 16 16:59:35.180907 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.180869 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct"] Jul 16 16:59:35.193409 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.193385 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh"] Jul 16 16:59:35.193603 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.193580 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" podUID="c0929e19-dae5-452a-a636-4c61bdec2a8d" containerName="manager" containerID="cri-o://8a963ca0c131cc23102f115f319078ed2ea877424bb8a7781f7e0829edc8ed61" gracePeriod=2 Jul 16 16:59:35.199298 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.199270 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh"] Jul 16 16:59:35.208161 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:59:35.208075 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pbxct\" is forbidden: User \"system:node:ip-10-0-136-129.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-129.ec2.internal' and this object" podUID="4885f2d3-f2af-4c60-b452-e1fa9d853d69" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:35.210440 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.210415 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8"] Jul 16 16:59:35.210734 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.210716 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0929e19-dae5-452a-a636-4c61bdec2a8d" containerName="manager" Jul 16 16:59:35.210734 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.210731 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0929e19-dae5-452a-a636-4c61bdec2a8d" containerName="manager" Jul 16 16:59:35.210899 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.210747 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4885f2d3-f2af-4c60-b452-e1fa9d853d69" containerName="manager" Jul 16 16:59:35.210899 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.210753 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4885f2d3-f2af-4c60-b452-e1fa9d853d69" containerName="manager" Jul 16 16:59:35.210899 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.210807 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4885f2d3-f2af-4c60-b452-e1fa9d853d69" containerName="manager" Jul 16 16:59:35.210899 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.210818 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0929e19-dae5-452a-a636-4c61bdec2a8d" containerName="manager" Jul 16 16:59:35.213583 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.213565 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8" Jul 16 16:59:35.232056 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.232030 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8"] Jul 16 16:59:35.236483 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.236453 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6z72\" (UniqueName: \"kubernetes.io/projected/3e834753-2f8c-4cf0-82ee-44625990a51c-kube-api-access-s6z72\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tm2g8\" (UID: \"3e834753-2f8c-4cf0-82ee-44625990a51c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8" Jul 16 16:59:35.236587 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.236557 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e834753-2f8c-4cf0-82ee-44625990a51c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tm2g8\" (UID: \"3e834753-2f8c-4cf0-82ee-44625990a51c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8" Jul 16 16:59:35.237492 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.237464 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nq7hv"] Jul 16 16:59:35.240670 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.240644 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nq7hv" Jul 16 16:59:35.259784 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.259758 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nq7hv"] Jul 16 16:59:35.275522 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:59:35.275491 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n2prh\" is forbidden: User \"system:node:ip-10-0-136-129.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-129.ec2.internal' and this object" podUID="c0929e19-dae5-452a-a636-4c61bdec2a8d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" Jul 16 16:59:35.277891 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:59:35.277861 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pbxct\" is forbidden: User \"system:node:ip-10-0-136-129.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-129.ec2.internal' and this object" podUID="4885f2d3-f2af-4c60-b452-e1fa9d853d69" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:35.280210 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:59:35.280186 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n2prh\" is forbidden: User \"system:node:ip-10-0-136-129.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-129.ec2.internal' and this object" podUID="c0929e19-dae5-452a-a636-4c61bdec2a8d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" Jul 16 16:59:35.304362 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:59:35.304332 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pbxct\" is forbidden: User \"system:node:ip-10-0-136-129.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-129.ec2.internal' and this object" podUID="4885f2d3-f2af-4c60-b452-e1fa9d853d69" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:35.337643 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.337601 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e834753-2f8c-4cf0-82ee-44625990a51c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tm2g8\" (UID: \"3e834753-2f8c-4cf0-82ee-44625990a51c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8" Jul 16 16:59:35.337791 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.337689 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwhwt\" (UniqueName: \"kubernetes.io/projected/f98619a8-c497-4579-90f9-e2561701d391-kube-api-access-qwhwt\") pod \"limitador-operator-controller-manager-859d7b78bd-nq7hv\" (UID: \"f98619a8-c497-4579-90f9-e2561701d391\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nq7hv" Jul 16 16:59:35.337911 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.337883 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6z72\" (UniqueName: \"kubernetes.io/projected/3e834753-2f8c-4cf0-82ee-44625990a51c-kube-api-access-s6z72\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tm2g8\" (UID: \"3e834753-2f8c-4cf0-82ee-44625990a51c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8" Jul 16 16:59:35.338215 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.338174 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e834753-2f8c-4cf0-82ee-44625990a51c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tm2g8\" (UID: \"3e834753-2f8c-4cf0-82ee-44625990a51c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8" Jul 16 16:59:35.349692 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.349663 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6z72\" (UniqueName: \"kubernetes.io/projected/3e834753-2f8c-4cf0-82ee-44625990a51c-kube-api-access-s6z72\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tm2g8\" (UID: \"3e834753-2f8c-4cf0-82ee-44625990a51c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8" Jul 16 16:59:35.436999 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.436974 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:35.438948 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.438927 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpzxt\" (UniqueName: \"kubernetes.io/projected/4885f2d3-f2af-4c60-b452-e1fa9d853d69-kube-api-access-zpzxt\") pod \"4885f2d3-f2af-4c60-b452-e1fa9d853d69\" (UID: \"4885f2d3-f2af-4c60-b452-e1fa9d853d69\") " Jul 16 16:59:35.439045 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.438963 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4885f2d3-f2af-4c60-b452-e1fa9d853d69-extensions-socket-volume\") pod \"4885f2d3-f2af-4c60-b452-e1fa9d853d69\" (UID: \"4885f2d3-f2af-4c60-b452-e1fa9d853d69\") " Jul 16 16:59:35.439147 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.439071 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwhwt\" (UniqueName: \"kubernetes.io/projected/f98619a8-c497-4579-90f9-e2561701d391-kube-api-access-qwhwt\") pod \"limitador-operator-controller-manager-859d7b78bd-nq7hv\" (UID: \"f98619a8-c497-4579-90f9-e2561701d391\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nq7hv" Jul 16 16:59:35.439520 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.439489 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4885f2d3-f2af-4c60-b452-e1fa9d853d69-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4885f2d3-f2af-4c60-b452-e1fa9d853d69" (UID: "4885f2d3-f2af-4c60-b452-e1fa9d853d69"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:59:35.439676 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.439660 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" Jul 16 16:59:35.440943 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.440925 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4885f2d3-f2af-4c60-b452-e1fa9d853d69-kube-api-access-zpzxt" (OuterVolumeSpecName: "kube-api-access-zpzxt") pod "4885f2d3-f2af-4c60-b452-e1fa9d853d69" (UID: "4885f2d3-f2af-4c60-b452-e1fa9d853d69"). InnerVolumeSpecName "kube-api-access-zpzxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:59:35.441016 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:59:35.440923 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n2prh\" is forbidden: User \"system:node:ip-10-0-136-129.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-129.ec2.internal' and this object" podUID="c0929e19-dae5-452a-a636-4c61bdec2a8d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" Jul 16 16:59:35.443292 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:59:35.443268 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pbxct\" is forbidden: User \"system:node:ip-10-0-136-129.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-129.ec2.internal' and this object" podUID="4885f2d3-f2af-4c60-b452-e1fa9d853d69" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:35.447414 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:59:35.447394 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pbxct\" is forbidden: User \"system:node:ip-10-0-136-129.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-129.ec2.internal' and this object" podUID="4885f2d3-f2af-4c60-b452-e1fa9d853d69" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:35.452592 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:59:35.452572 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n2prh\" is forbidden: User \"system:node:ip-10-0-136-129.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-129.ec2.internal' and this object" podUID="c0929e19-dae5-452a-a636-4c61bdec2a8d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" Jul 16 16:59:35.456683 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.456664 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwhwt\" (UniqueName: \"kubernetes.io/projected/f98619a8-c497-4579-90f9-e2561701d391-kube-api-access-qwhwt\") pod \"limitador-operator-controller-manager-859d7b78bd-nq7hv\" (UID: \"f98619a8-c497-4579-90f9-e2561701d391\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nq7hv" Jul 16 16:59:35.539841 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.539808 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jk4n6\" (UniqueName: \"kubernetes.io/projected/c0929e19-dae5-452a-a636-4c61bdec2a8d-kube-api-access-jk4n6\") pod \"c0929e19-dae5-452a-a636-4c61bdec2a8d\" (UID: \"c0929e19-dae5-452a-a636-4c61bdec2a8d\") " Jul 16 16:59:35.539996 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.539953 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpzxt\" (UniqueName: \"kubernetes.io/projected/4885f2d3-f2af-4c60-b452-e1fa9d853d69-kube-api-access-zpzxt\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:59:35.539996 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.539964 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4885f2d3-f2af-4c60-b452-e1fa9d853d69-extensions-socket-volume\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:59:35.541922 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.541896 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0929e19-dae5-452a-a636-4c61bdec2a8d-kube-api-access-jk4n6" (OuterVolumeSpecName: "kube-api-access-jk4n6") pod "c0929e19-dae5-452a-a636-4c61bdec2a8d" (UID: "c0929e19-dae5-452a-a636-4c61bdec2a8d"). InnerVolumeSpecName "kube-api-access-jk4n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:59:35.556079 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.556055 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:35.591596 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.591574 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8" Jul 16 16:59:35.597690 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.597666 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nq7hv" Jul 16 16:59:35.640750 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.640708 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk4n6\" (UniqueName: \"kubernetes.io/projected/c0929e19-dae5-452a-a636-4c61bdec2a8d-kube-api-access-jk4n6\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:59:35.711065 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.711037 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:35.778699 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:59:35.778673 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e834753_2f8c_4cf0_82ee_44625990a51c.slice/crio-223a792b9e13a17ab5c6eef064c3405a0fa8e5c2f0433fe96258e6c0180261a4 WatchSource:0}: Error finding container 223a792b9e13a17ab5c6eef064c3405a0fa8e5c2f0433fe96258e6c0180261a4: Status 404 returned error can't find the container with id 223a792b9e13a17ab5c6eef064c3405a0fa8e5c2f0433fe96258e6c0180261a4 Jul 16 16:59:35.785815 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.785788 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8"] Jul 16 16:59:35.813796 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.811855 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nq7hv"] Jul 16 16:59:35.840363 ip-10-0-136-129 kubenswrapper[2575]: W0716 16:59:35.840323 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98619a8_c497_4579_90f9_e2561701d391.slice/crio-ebf65f0b086b4e81e7567cecae9371e1d5d0ba7d7cdcc8e856adc48b74a4eb79 WatchSource:0}: Error finding container ebf65f0b086b4e81e7567cecae9371e1d5d0ba7d7cdcc8e856adc48b74a4eb79: Status 404 returned error can't find the container with id ebf65f0b086b4e81e7567cecae9371e1d5d0ba7d7cdcc8e856adc48b74a4eb79 Jul 16 16:59:35.873308 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.873280 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:35.903170 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.903140 2575 generic.go:356] "Generic (PLEG): container finished" podID="4885f2d3-f2af-4c60-b452-e1fa9d853d69" containerID="5116c63b3c8258f1e946e88debb46a99fa525d492d9555d45fdc1c4d3729b0c4" exitCode=0 Jul 16 16:59:35.903273 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.903228 2575 scope.go:117] "RemoveContainer" containerID="5116c63b3c8258f1e946e88debb46a99fa525d492d9555d45fdc1c4d3729b0c4" Jul 16 16:59:35.903335 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.903314 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:35.904678 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.904642 2575 generic.go:356] "Generic (PLEG): container finished" podID="c0929e19-dae5-452a-a636-4c61bdec2a8d" containerID="8a963ca0c131cc23102f115f319078ed2ea877424bb8a7781f7e0829edc8ed61" exitCode=0 Jul 16 16:59:35.904786 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.904700 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" Jul 16 16:59:35.906456 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.906435 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nq7hv" event={"ID":"f98619a8-c497-4579-90f9-e2561701d391","Type":"ContainerStarted","Data":"6450486b95f90a3605ff66ccb0d2f9cd613a561ebd0943a9d4a4a48cf243779e"} Jul 16 16:59:35.906456 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:59:35.906441 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n2prh\" is forbidden: User \"system:node:ip-10-0-136-129.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-129.ec2.internal' and this object" podUID="c0929e19-dae5-452a-a636-4c61bdec2a8d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" Jul 16 16:59:35.906626 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.906482 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nq7hv" event={"ID":"f98619a8-c497-4579-90f9-e2561701d391","Type":"ContainerStarted","Data":"ebf65f0b086b4e81e7567cecae9371e1d5d0ba7d7cdcc8e856adc48b74a4eb79"} Jul 16 16:59:35.906626 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.906588 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nq7hv" Jul 16 16:59:35.908455 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.908430 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8" event={"ID":"3e834753-2f8c-4cf0-82ee-44625990a51c","Type":"ContainerStarted","Data":"6996b5ed9eb32ab3ca0900c2d4747a118263183385626b3456c3cf566e6fdf29"} Jul 16 16:59:35.908569 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.908477 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8" event={"ID":"3e834753-2f8c-4cf0-82ee-44625990a51c","Type":"ContainerStarted","Data":"223a792b9e13a17ab5c6eef064c3405a0fa8e5c2f0433fe96258e6c0180261a4"} Jul 16 16:59:35.908569 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.908511 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8" Jul 16 16:59:35.912192 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.912170 2575 scope.go:117] "RemoveContainer" containerID="5116c63b3c8258f1e946e88debb46a99fa525d492d9555d45fdc1c4d3729b0c4" Jul 16 16:59:35.912502 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:59:35.912481 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5116c63b3c8258f1e946e88debb46a99fa525d492d9555d45fdc1c4d3729b0c4\": container with ID starting with 5116c63b3c8258f1e946e88debb46a99fa525d492d9555d45fdc1c4d3729b0c4 not found: ID does not exist" containerID="5116c63b3c8258f1e946e88debb46a99fa525d492d9555d45fdc1c4d3729b0c4" Jul 16 16:59:35.912593 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.912512 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5116c63b3c8258f1e946e88debb46a99fa525d492d9555d45fdc1c4d3729b0c4"} err="failed to get container status \"5116c63b3c8258f1e946e88debb46a99fa525d492d9555d45fdc1c4d3729b0c4\": rpc error: code = NotFound desc = could not find container \"5116c63b3c8258f1e946e88debb46a99fa525d492d9555d45fdc1c4d3729b0c4\": container with ID starting with 5116c63b3c8258f1e946e88debb46a99fa525d492d9555d45fdc1c4d3729b0c4 not found: ID does not exist" Jul 16 16:59:35.912593 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.912535 2575 scope.go:117] "RemoveContainer" containerID="8a963ca0c131cc23102f115f319078ed2ea877424bb8a7781f7e0829edc8ed61" Jul 16 16:59:35.914146 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:59:35.914073 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pbxct\" is forbidden: User \"system:node:ip-10-0-136-129.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-129.ec2.internal' and this object" podUID="4885f2d3-f2af-4c60-b452-e1fa9d853d69" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:35.916610 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:59:35.916588 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n2prh\" is forbidden: User \"system:node:ip-10-0-136-129.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-129.ec2.internal' and this object" podUID="c0929e19-dae5-452a-a636-4c61bdec2a8d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" Jul 16 16:59:35.920002 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.919986 2575 scope.go:117] "RemoveContainer" containerID="8a963ca0c131cc23102f115f319078ed2ea877424bb8a7781f7e0829edc8ed61" Jul 16 16:59:35.920315 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:59:35.920295 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a963ca0c131cc23102f115f319078ed2ea877424bb8a7781f7e0829edc8ed61\": container with ID starting with 8a963ca0c131cc23102f115f319078ed2ea877424bb8a7781f7e0829edc8ed61 not found: ID does not exist" containerID="8a963ca0c131cc23102f115f319078ed2ea877424bb8a7781f7e0829edc8ed61" Jul 16 16:59:35.920387 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.920325 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a963ca0c131cc23102f115f319078ed2ea877424bb8a7781f7e0829edc8ed61"} err="failed to get container status \"8a963ca0c131cc23102f115f319078ed2ea877424bb8a7781f7e0829edc8ed61\": rpc error: code = NotFound desc = could not find container \"8a963ca0c131cc23102f115f319078ed2ea877424bb8a7781f7e0829edc8ed61\": container with ID starting with 8a963ca0c131cc23102f115f319078ed2ea877424bb8a7781f7e0829edc8ed61 not found: ID does not exist" Jul 16 16:59:35.948849 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:35.948804 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nq7hv" podStartSLOduration=0.948788667 podStartE2EDuration="948.788667ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:59:35.9472567 +0000 UTC m=+513.465981935" watchObservedRunningTime="2026-07-16 16:59:35.948788667 +0000 UTC m=+513.467513900" Jul 16 16:59:35.950130 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:59:35.950079 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pbxct\" is forbidden: User \"system:node:ip-10-0-136-129.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-129.ec2.internal' and this object" podUID="4885f2d3-f2af-4c60-b452-e1fa9d853d69" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:36.037053 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:36.036948 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8" podStartSLOduration=1.036932679 podStartE2EDuration="1.036932679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:59:36.036348396 +0000 UTC m=+513.555073634" watchObservedRunningTime="2026-07-16 16:59:36.036932679 +0000 UTC m=+513.555657912" Jul 16 16:59:36.039649 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:59:36.039616 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-n2prh\" is forbidden: User \"system:node:ip-10-0-136-129.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-129.ec2.internal' and this object" podUID="c0929e19-dae5-452a-a636-4c61bdec2a8d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-n2prh" Jul 16 16:59:36.044550 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:59:36.044518 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-pbxct\" is forbidden: User \"system:node:ip-10-0-136-129.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-129.ec2.internal' and this object" podUID="4885f2d3-f2af-4c60-b452-e1fa9d853d69" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-pbxct" Jul 16 16:59:36.460320 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:36.460286 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:36.908592 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:36.908561 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4885f2d3-f2af-4c60-b452-e1fa9d853d69" path="/var/lib/kubelet/pods/4885f2d3-f2af-4c60-b452-e1fa9d853d69/volumes" Jul 16 16:59:36.908947 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:36.908878 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0929e19-dae5-452a-a636-4c61bdec2a8d" path="/var/lib/kubelet/pods/c0929e19-dae5-452a-a636-4c61bdec2a8d/volumes" Jul 16 16:59:38.451129 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:38.451084 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:38.540306 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:38.540275 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:41.130480 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:41.130438 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:41.251296 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:41.251263 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:44.879560 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:44.879521 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:44.966822 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:44.966787 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:46.915533 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:46.915505 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8" Jul 16 16:59:46.915961 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:46.915574 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nq7hv" Jul 16 16:59:53.308014 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:53.307970 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8"] Jul 16 16:59:53.308496 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:53.308232 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8" podUID="3e834753-2f8c-4cf0-82ee-44625990a51c" containerName="manager" containerID="cri-o://6996b5ed9eb32ab3ca0900c2d4747a118263183385626b3456c3cf566e6fdf29" gracePeriod=10 Jul 16 16:59:53.560772 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:53.560707 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8" Jul 16 16:59:53.570043 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:53.570021 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e834753-2f8c-4cf0-82ee-44625990a51c-extensions-socket-volume\") pod \"3e834753-2f8c-4cf0-82ee-44625990a51c\" (UID: \"3e834753-2f8c-4cf0-82ee-44625990a51c\") " Jul 16 16:59:53.570180 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:53.570077 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6z72\" (UniqueName: \"kubernetes.io/projected/3e834753-2f8c-4cf0-82ee-44625990a51c-kube-api-access-s6z72\") pod \"3e834753-2f8c-4cf0-82ee-44625990a51c\" (UID: \"3e834753-2f8c-4cf0-82ee-44625990a51c\") " Jul 16 16:59:53.570386 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:53.570364 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e834753-2f8c-4cf0-82ee-44625990a51c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3e834753-2f8c-4cf0-82ee-44625990a51c" (UID: "3e834753-2f8c-4cf0-82ee-44625990a51c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:59:53.572062 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:53.572042 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e834753-2f8c-4cf0-82ee-44625990a51c-kube-api-access-s6z72" (OuterVolumeSpecName: "kube-api-access-s6z72") pod "3e834753-2f8c-4cf0-82ee-44625990a51c" (UID: "3e834753-2f8c-4cf0-82ee-44625990a51c"). InnerVolumeSpecName "kube-api-access-s6z72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:59:53.670793 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:53.670753 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e834753-2f8c-4cf0-82ee-44625990a51c-extensions-socket-volume\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:59:53.670793 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:53.670783 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6z72\" (UniqueName: \"kubernetes.io/projected/3e834753-2f8c-4cf0-82ee-44625990a51c-kube-api-access-s6z72\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 16:59:53.988356 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:53.988323 2575 generic.go:356] "Generic (PLEG): container finished" podID="3e834753-2f8c-4cf0-82ee-44625990a51c" containerID="6996b5ed9eb32ab3ca0900c2d4747a118263183385626b3456c3cf566e6fdf29" exitCode=0 Jul 16 16:59:53.988545 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:53.988363 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8" event={"ID":"3e834753-2f8c-4cf0-82ee-44625990a51c","Type":"ContainerDied","Data":"6996b5ed9eb32ab3ca0900c2d4747a118263183385626b3456c3cf566e6fdf29"} Jul 16 16:59:53.988545 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:53.988399 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8" event={"ID":"3e834753-2f8c-4cf0-82ee-44625990a51c","Type":"ContainerDied","Data":"223a792b9e13a17ab5c6eef064c3405a0fa8e5c2f0433fe96258e6c0180261a4"} Jul 16 16:59:53.988545 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:53.988419 2575 scope.go:117] "RemoveContainer" containerID="6996b5ed9eb32ab3ca0900c2d4747a118263183385626b3456c3cf566e6fdf29" Jul 16 16:59:53.988545 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:53.988364 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8" Jul 16 16:59:53.996936 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:53.996916 2575 scope.go:117] "RemoveContainer" containerID="6996b5ed9eb32ab3ca0900c2d4747a118263183385626b3456c3cf566e6fdf29" Jul 16 16:59:53.997223 ip-10-0-136-129 kubenswrapper[2575]: E0716 16:59:53.997202 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6996b5ed9eb32ab3ca0900c2d4747a118263183385626b3456c3cf566e6fdf29\": container with ID starting with 6996b5ed9eb32ab3ca0900c2d4747a118263183385626b3456c3cf566e6fdf29 not found: ID does not exist" containerID="6996b5ed9eb32ab3ca0900c2d4747a118263183385626b3456c3cf566e6fdf29" Jul 16 16:59:53.997304 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:53.997230 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6996b5ed9eb32ab3ca0900c2d4747a118263183385626b3456c3cf566e6fdf29"} err="failed to get container status \"6996b5ed9eb32ab3ca0900c2d4747a118263183385626b3456c3cf566e6fdf29\": rpc error: code = NotFound desc = could not find container \"6996b5ed9eb32ab3ca0900c2d4747a118263183385626b3456c3cf566e6fdf29\": container with ID starting with 6996b5ed9eb32ab3ca0900c2d4747a118263183385626b3456c3cf566e6fdf29 not found: ID does not exist" Jul 16 16:59:54.120063 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:54.120023 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8"] Jul 16 16:59:54.121324 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:54.121306 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tm2g8"] Jul 16 16:59:54.909063 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:54.908989 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e834753-2f8c-4cf0-82ee-44625990a51c" path="/var/lib/kubelet/pods/3e834753-2f8c-4cf0-82ee-44625990a51c/volumes" Jul 16 16:59:55.080167 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:55.080129 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:55.181709 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:55.181589 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:55.309298 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:55.309228 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:55.418584 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:55.418548 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:58.455710 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:58.455640 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 16:59:58.556299 ip-10-0-136-129 kubenswrapper[2575]: I0716 16:59:58.556264 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:10.408850 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.408818 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm"] Jul 16 17:00:10.409316 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.409280 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e834753-2f8c-4cf0-82ee-44625990a51c" containerName="manager" Jul 16 17:00:10.409316 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.409298 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e834753-2f8c-4cf0-82ee-44625990a51c" containerName="manager" Jul 16 17:00:10.409420 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.409392 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e834753-2f8c-4cf0-82ee-44625990a51c" containerName="manager" Jul 16 17:00:10.414437 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.414414 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.417848 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.417819 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7s7x6\"" Jul 16 17:00:10.434029 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.434001 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm"] Jul 16 17:00:10.439726 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.439698 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:10.524181 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.524151 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.524359 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.524190 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.524359 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.524224 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdltm\" (UniqueName: \"kubernetes.io/projected/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-kube-api-access-hdltm\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.524359 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.524292 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.524359 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.524342 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.524507 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.524363 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.524507 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.524411 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.524507 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.524439 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.524507 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.524496 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.558682 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.558648 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:10.626105 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.626051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.626105 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.626087 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.626364 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.626135 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.626364 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.626158 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.626364 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.626183 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.626364 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.626235 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.626364 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.626269 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.626615 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.626538 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdltm\" (UniqueName: \"kubernetes.io/projected/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-kube-api-access-hdltm\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.626615 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.626589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.626615 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.626594 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.626840 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.626820 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.626956 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.626909 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.626956 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.626842 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.627504 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.627481 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.628588 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.628566 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.629156 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.629135 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.639076 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.639040 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.639254 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.639223 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdltm\" (UniqueName: \"kubernetes.io/projected/9b6eccbb-8a9b-46b8-864b-001193c5f1fd-kube-api-access-hdltm\") pod \"maas-default-gateway-openshift-default-56654dcf7-sckjm\" (UID: \"9b6eccbb-8a9b-46b8-864b-001193c5f1fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.728461 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.728375 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:10.865813 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.865786 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm"] Jul 16 17:00:10.867501 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:00:10.867472 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6eccbb_8a9b_46b8_864b_001193c5f1fd.slice/crio-25a1c8e051d93898a0d95f395fc7d3d35b01b02575e592b7bd60b75a40856897 WatchSource:0}: Error finding container 25a1c8e051d93898a0d95f395fc7d3d35b01b02575e592b7bd60b75a40856897: Status 404 returned error can't find the container with id 25a1c8e051d93898a0d95f395fc7d3d35b01b02575e592b7bd60b75a40856897 Jul 16 17:00:10.869472 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.869439 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:00:10.869562 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.869508 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:00:10.869562 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:10.869540 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:00:11.059172 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:11.059059 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" event={"ID":"9b6eccbb-8a9b-46b8-864b-001193c5f1fd","Type":"ContainerStarted","Data":"0e9303760bc6d1053efcad5b51240643b4c11f8586aef27cc84ec4e3a60f4424"} Jul 16 17:00:11.059172 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:11.059129 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" event={"ID":"9b6eccbb-8a9b-46b8-864b-001193c5f1fd","Type":"ContainerStarted","Data":"25a1c8e051d93898a0d95f395fc7d3d35b01b02575e592b7bd60b75a40856897"} Jul 16 17:00:11.085851 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:11.085778 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" podStartSLOduration=1.085761522 podStartE2EDuration="1.085761522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:00:11.083981072 +0000 UTC m=+548.602706305" watchObservedRunningTime="2026-07-16 17:00:11.085761522 +0000 UTC m=+548.604486865" Jul 16 17:00:11.728930 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:11.728884 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:11.733425 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:11.733401 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:12.063289 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:12.063202 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:12.064289 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:12.064271 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:00:12.184995 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:12.184961 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:12.339162 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:12.339073 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:12.440982 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:12.440952 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:12.686544 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:12.686513 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:00:12.689888 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:12.689870 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-lbtrw" Jul 16 17:00:12.696553 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:12.696524 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-chkj9\"" Jul 16 17:00:12.697272 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:12.697142 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 16 17:00:12.704159 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:12.704089 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:00:12.739447 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:12.739410 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:00:12.845770 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:12.845734 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b89814e0-9771-4f35-8798-36ab391e9603-config-file\") pod \"limitador-limitador-54d6d988d5-lbtrw\" (UID: \"b89814e0-9771-4f35-8798-36ab391e9603\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lbtrw" Jul 16 17:00:12.845941 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:12.845813 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjh9h\" (UniqueName: \"kubernetes.io/projected/b89814e0-9771-4f35-8798-36ab391e9603-kube-api-access-zjh9h\") pod \"limitador-limitador-54d6d988d5-lbtrw\" (UID: \"b89814e0-9771-4f35-8798-36ab391e9603\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lbtrw" Jul 16 17:00:12.946585 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:12.946498 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjh9h\" (UniqueName: \"kubernetes.io/projected/b89814e0-9771-4f35-8798-36ab391e9603-kube-api-access-zjh9h\") pod \"limitador-limitador-54d6d988d5-lbtrw\" (UID: \"b89814e0-9771-4f35-8798-36ab391e9603\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lbtrw" Jul 16 17:00:12.946585 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:12.946566 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b89814e0-9771-4f35-8798-36ab391e9603-config-file\") pod \"limitador-limitador-54d6d988d5-lbtrw\" (UID: \"b89814e0-9771-4f35-8798-36ab391e9603\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lbtrw" Jul 16 17:00:12.947216 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:12.947200 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b89814e0-9771-4f35-8798-36ab391e9603-config-file\") pod \"limitador-limitador-54d6d988d5-lbtrw\" (UID: \"b89814e0-9771-4f35-8798-36ab391e9603\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lbtrw" Jul 16 17:00:12.961248 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:12.961219 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjh9h\" (UniqueName: \"kubernetes.io/projected/b89814e0-9771-4f35-8798-36ab391e9603-kube-api-access-zjh9h\") pod \"limitador-limitador-54d6d988d5-lbtrw\" (UID: \"b89814e0-9771-4f35-8798-36ab391e9603\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lbtrw" Jul 16 17:00:13.000231 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:13.000197 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-lbtrw" Jul 16 17:00:13.140900 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:13.140748 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:00:13.143583 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:00:13.143553 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89814e0_9771_4f35_8798_36ab391e9603.slice/crio-a0c240fe3657fd24c4e80d6da20a09a1c331d62e84b3d123259c092488b65d6a WatchSource:0}: Error finding container a0c240fe3657fd24c4e80d6da20a09a1c331d62e84b3d123259c092488b65d6a: Status 404 returned error can't find the container with id a0c240fe3657fd24c4e80d6da20a09a1c331d62e84b3d123259c092488b65d6a Jul 16 17:00:13.215483 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:13.215399 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:13.320706 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:13.320681 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:14.076338 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:14.076299 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-lbtrw" event={"ID":"b89814e0-9771-4f35-8798-36ab391e9603","Type":"ContainerStarted","Data":"a0c240fe3657fd24c4e80d6da20a09a1c331d62e84b3d123259c092488b65d6a"} Jul 16 17:00:16.087544 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:16.087512 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-lbtrw" event={"ID":"b89814e0-9771-4f35-8798-36ab391e9603","Type":"ContainerStarted","Data":"418bf8864ddc60d69069fb5a78c08f36250bbb1bf5debec3e12e4e1dd1f68322"} Jul 16 17:00:16.087967 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:16.087584 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-lbtrw" Jul 16 17:00:16.111769 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:16.111714 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-lbtrw" podStartSLOduration=1.3414352219999999 podStartE2EDuration="4.111699225s" totalImagesPullingTime="2.770264003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:00:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:00:16.107262425 +0000 UTC m=+553.625987661" watchObservedRunningTime="2026-07-16 17:00:16.111699225 +0000 UTC m=+553.630424459" Jul 16 17:00:19.342107 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:19.342062 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:19.458712 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:19.458683 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:22.954585 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:22.954498 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:27.092083 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:27.092050 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-lbtrw" Jul 16 17:00:28.363073 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:28.363028 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:28.462144 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:28.462114 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:28.942245 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:28.942218 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:31.404041 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:31.404012 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:31.513319 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:31.513285 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:34.057684 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:34.057652 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:40.199569 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:40.199537 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:41.006293 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:41.006253 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:41.272131 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:41.272041 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:48.678651 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:48.678613 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:48.803593 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:48.803562 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:48.971046 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:48.970936 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:52.083341 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:52.083293 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:00:52.177217 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:00:52.177157 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:01:02.845752 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:02.845724 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 17:01:02.848080 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:02.848061 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 17:01:03.443041 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:03.443010 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:01:03.524752 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:03.524720 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:01:23.786935 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:23.786903 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:01:25.793324 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:25.793246 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:01:25.835600 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:25.835564 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-bblpd"] Jul 16 17:01:25.838782 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:25.838759 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-bblpd" Jul 16 17:01:25.846794 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:25.846773 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 16 17:01:25.846902 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:25.846836 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l9xhj\"" Jul 16 17:01:25.876765 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:25.876732 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcdxl\" (UniqueName: \"kubernetes.io/projected/009da2f0-0152-4212-9e02-404799d12456-kube-api-access-kcdxl\") pod \"payload-processing-f9b8d9b9b-bblpd\" (UID: \"009da2f0-0152-4212-9e02-404799d12456\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-bblpd" Jul 16 17:01:25.877158 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:25.877130 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/009da2f0-0152-4212-9e02-404799d12456-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-bblpd\" (UID: \"009da2f0-0152-4212-9e02-404799d12456\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-bblpd" Jul 16 17:01:25.877795 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:25.877769 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-bblpd"] Jul 16 17:01:25.978312 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:25.978283 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/009da2f0-0152-4212-9e02-404799d12456-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-bblpd\" (UID: \"009da2f0-0152-4212-9e02-404799d12456\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-bblpd" Jul 16 17:01:25.978480 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:25.978456 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcdxl\" (UniqueName: \"kubernetes.io/projected/009da2f0-0152-4212-9e02-404799d12456-kube-api-access-kcdxl\") pod \"payload-processing-f9b8d9b9b-bblpd\" (UID: \"009da2f0-0152-4212-9e02-404799d12456\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-bblpd" Jul 16 17:01:25.978955 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:25.978933 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/009da2f0-0152-4212-9e02-404799d12456-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-bblpd\" (UID: \"009da2f0-0152-4212-9e02-404799d12456\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-bblpd" Jul 16 17:01:26.001876 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:26.001845 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcdxl\" (UniqueName: \"kubernetes.io/projected/009da2f0-0152-4212-9e02-404799d12456-kube-api-access-kcdxl\") pod \"payload-processing-f9b8d9b9b-bblpd\" (UID: \"009da2f0-0152-4212-9e02-404799d12456\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-bblpd" Jul 16 17:01:26.148518 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:26.148473 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-bblpd" Jul 16 17:01:26.307686 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:26.307653 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-bblpd"] Jul 16 17:01:26.310140 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:01:26.310115 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009da2f0_0152_4212_9e02_404799d12456.slice/crio-2592249c0de4110afa74b4688585b1ce8745440a56de4aba5ecf1331aa317a9e WatchSource:0}: Error finding container 2592249c0de4110afa74b4688585b1ce8745440a56de4aba5ecf1331aa317a9e: Status 404 returned error can't find the container with id 2592249c0de4110afa74b4688585b1ce8745440a56de4aba5ecf1331aa317a9e Jul 16 17:01:26.383666 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:26.383631 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-bblpd" event={"ID":"009da2f0-0152-4212-9e02-404799d12456","Type":"ContainerStarted","Data":"2592249c0de4110afa74b4688585b1ce8745440a56de4aba5ecf1331aa317a9e"} Jul 16 17:01:28.394017 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:28.393985 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-bblpd" event={"ID":"009da2f0-0152-4212-9e02-404799d12456","Type":"ContainerStarted","Data":"b7059ede06f8f29e80aeab351c0692f432a43cf613258f88086be7a1889ec7af"} Jul 16 17:01:28.394406 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:28.394047 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-bblpd" Jul 16 17:01:28.420653 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:28.420604 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-bblpd" podStartSLOduration=1.847798566 podStartE2EDuration="3.420586998s" totalImagesPullingTime="1.572788432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:01:28.417878789 +0000 UTC m=+625.936604024" watchObservedRunningTime="2026-07-16 17:01:28.420586998 +0000 UTC m=+625.939312232" Jul 16 17:01:33.073111 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:33.073073 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:01:33.146906 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:01:33.146875 2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6eccbb_8a9b_46b8_864b_001193c5f1fd.slice/crio-0e9303760bc6d1053efcad5b51240643b4c11f8586aef27cc84ec4e3a60f4424.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@0-12595-0.service\": RecentStats: unable to find data in memory cache]" Jul 16 17:01:33.415348 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:33.415312 2575 generic.go:356] "Generic (PLEG): container finished" podID="9b6eccbb-8a9b-46b8-864b-001193c5f1fd" containerID="0e9303760bc6d1053efcad5b51240643b4c11f8586aef27cc84ec4e3a60f4424" exitCode=0 Jul 16 17:01:33.415514 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:33.415384 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" event={"ID":"9b6eccbb-8a9b-46b8-864b-001193c5f1fd","Type":"ContainerDied","Data":"0e9303760bc6d1053efcad5b51240643b4c11f8586aef27cc84ec4e3a60f4424"} Jul 16 17:01:33.415787 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:33.415771 2575 scope.go:117] "RemoveContainer" containerID="0e9303760bc6d1053efcad5b51240643b4c11f8586aef27cc84ec4e3a60f4424" Jul 16 17:01:33.416707 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:33.416681 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:01:33.416790 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:33.416744 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:01:33.416790 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:33.416773 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:01:34.421308 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:34.421274 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" event={"ID":"9b6eccbb-8a9b-46b8-864b-001193c5f1fd","Type":"ContainerStarted","Data":"346575d0bbd6d630f16053978a71f33cfb3ec2527f659fcc12235112227177ab"} Jul 16 17:01:34.729586 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:34.729018 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:01:34.729586 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:34.729177 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-sckjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 16 17:01:34.729586 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:34.729225 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" podUID="9b6eccbb-8a9b-46b8-864b-001193c5f1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 16 17:01:35.074399 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:35.074298 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:01:35.163211 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:35.163175 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:01:35.729423 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:35.728958 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-sckjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 16 17:01:35.729423 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:35.729020 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" podUID="9b6eccbb-8a9b-46b8-864b-001193c5f1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 16 17:01:36.728880 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:36.728844 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-sckjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 16 17:01:36.729604 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:36.729178 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" podUID="9b6eccbb-8a9b-46b8-864b-001193c5f1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 16 17:01:37.729047 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:37.729012 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-sckjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 16 17:01:37.729364 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:37.729336 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" podUID="9b6eccbb-8a9b-46b8-864b-001193c5f1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 16 17:01:38.732639 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:38.732610 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:01:38.733021 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:38.732935 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:01:38.733739 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:38.733719 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-sckjm" Jul 16 17:01:39.400159 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:39.400123 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-bblpd" Jul 16 17:01:41.100306 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:41.100271 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:01:41.542302 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:41.542269 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:01:47.415916 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:47.415873 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:01:48.278589 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:48.278555 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:01:52.270646 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.270607 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7"] Jul 16 17:01:52.276397 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.276380 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.280742 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.280717 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 16 17:01:52.280864 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.280747 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 17:01:52.280864 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.280783 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tgg58\"" Jul 16 17:01:52.280864 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.280825 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 17:01:52.291365 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.291336 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7"] Jul 16 17:01:52.321416 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.321385 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:01:52.408723 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.408693 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1603bec9-df29-4d1a-b070-29fafa963ffe-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4frr7\" (UID: \"1603bec9-df29-4d1a-b070-29fafa963ffe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.408916 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.408749 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1603bec9-df29-4d1a-b070-29fafa963ffe-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4frr7\" (UID: \"1603bec9-df29-4d1a-b070-29fafa963ffe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.408916 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.408814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1603bec9-df29-4d1a-b070-29fafa963ffe-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4frr7\" (UID: \"1603bec9-df29-4d1a-b070-29fafa963ffe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.408916 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.408869 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1603bec9-df29-4d1a-b070-29fafa963ffe-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4frr7\" (UID: \"1603bec9-df29-4d1a-b070-29fafa963ffe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.409049 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.408922 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1603bec9-df29-4d1a-b070-29fafa963ffe-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4frr7\" (UID: \"1603bec9-df29-4d1a-b070-29fafa963ffe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.409049 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.408948 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmfnk\" (UniqueName: \"kubernetes.io/projected/1603bec9-df29-4d1a-b070-29fafa963ffe-kube-api-access-nmfnk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4frr7\" (UID: \"1603bec9-df29-4d1a-b070-29fafa963ffe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.510357 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.510324 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1603bec9-df29-4d1a-b070-29fafa963ffe-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4frr7\" (UID: \"1603bec9-df29-4d1a-b070-29fafa963ffe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.510357 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.510360 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1603bec9-df29-4d1a-b070-29fafa963ffe-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4frr7\" (UID: \"1603bec9-df29-4d1a-b070-29fafa963ffe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.510635 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.510381 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1603bec9-df29-4d1a-b070-29fafa963ffe-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4frr7\" (UID: \"1603bec9-df29-4d1a-b070-29fafa963ffe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.510635 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.510405 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1603bec9-df29-4d1a-b070-29fafa963ffe-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4frr7\" (UID: \"1603bec9-df29-4d1a-b070-29fafa963ffe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.510635 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.510421 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmfnk\" (UniqueName: \"kubernetes.io/projected/1603bec9-df29-4d1a-b070-29fafa963ffe-kube-api-access-nmfnk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4frr7\" (UID: \"1603bec9-df29-4d1a-b070-29fafa963ffe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.510635 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.510453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1603bec9-df29-4d1a-b070-29fafa963ffe-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4frr7\" (UID: \"1603bec9-df29-4d1a-b070-29fafa963ffe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.510846 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.510733 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1603bec9-df29-4d1a-b070-29fafa963ffe-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4frr7\" (UID: \"1603bec9-df29-4d1a-b070-29fafa963ffe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.510846 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.510779 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1603bec9-df29-4d1a-b070-29fafa963ffe-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4frr7\" (UID: \"1603bec9-df29-4d1a-b070-29fafa963ffe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.510936 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.510856 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1603bec9-df29-4d1a-b070-29fafa963ffe-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4frr7\" (UID: \"1603bec9-df29-4d1a-b070-29fafa963ffe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.512696 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.512667 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1603bec9-df29-4d1a-b070-29fafa963ffe-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4frr7\" (UID: \"1603bec9-df29-4d1a-b070-29fafa963ffe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.512940 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.512924 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1603bec9-df29-4d1a-b070-29fafa963ffe-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4frr7\" (UID: \"1603bec9-df29-4d1a-b070-29fafa963ffe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.521921 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.521869 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmfnk\" (UniqueName: \"kubernetes.io/projected/1603bec9-df29-4d1a-b070-29fafa963ffe-kube-api-access-nmfnk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4frr7\" (UID: \"1603bec9-df29-4d1a-b070-29fafa963ffe\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.586967 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.586933 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:01:52.734280 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:52.734241 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7"] Jul 16 17:01:52.736468 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:01:52.736441 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1603bec9_df29_4d1a_b070_29fafa963ffe.slice/crio-1b734d822f75e8a1c699cc86cc6eeff8360d59b1a7463c874a0d53559db5d156 WatchSource:0}: Error finding container 1b734d822f75e8a1c699cc86cc6eeff8360d59b1a7463c874a0d53559db5d156: Status 404 returned error can't find the container with id 1b734d822f75e8a1c699cc86cc6eeff8360d59b1a7463c874a0d53559db5d156 Jul 16 17:01:53.388471 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:53.388436 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:01:53.508758 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:53.508715 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" event={"ID":"1603bec9-df29-4d1a-b070-29fafa963ffe","Type":"ContainerStarted","Data":"1b734d822f75e8a1c699cc86cc6eeff8360d59b1a7463c874a0d53559db5d156"} Jul 16 17:01:54.514201 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:54.514162 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" event={"ID":"1603bec9-df29-4d1a-b070-29fafa963ffe","Type":"ContainerStarted","Data":"e5add790ede463db4d5c66dbf92b6729b33e6882b15aa23e915367c86a8b2fee"} Jul 16 17:01:54.555402 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:01:54.555349 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" podStartSLOduration=1.5394640480000001 podStartE2EDuration="2.555333317s" totalImagesPullingTime="1.015869269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:01:54.555039176 +0000 UTC m=+652.073764410" watchObservedRunningTime="2026-07-16 17:01:54.555333317 +0000 UTC m=+652.074058551" Jul 16 17:02:00.296582 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:02:00.296549 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:02:02.322263 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:02:02.322229 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:02:02.587500 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:02:02.587409 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:02:02.587500 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:02:02.587474 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:02:02.599794 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:02:02.599770 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:02:03.441911 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:02:03.441875 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:02:03.561261 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:02:03.561237 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4frr7" Jul 16 17:02:10.666838 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:02:10.666798 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:02:19.276979 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:02:19.276948 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:02:21.467087 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:02:21.467055 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:03:41.986948 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:03:41.986908 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:03:42.092941 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:03:42.092913 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:03:49.931883 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:03:49.931851 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:03:50.030203 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:03:50.030169 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:04:09.259110 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:04:09.259061 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:04:09.559010 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:04:09.558922 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:04:21.064489 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:04:21.064446 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:04:25.499568 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:04:25.499482 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:04:30.275951 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:04:30.275916 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:04:40.465605 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:04:40.465566 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:04:50.663641 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:04:50.663599 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:04:53.429618 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:04:53.429582 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:04:54.970509 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:04:54.970476 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:05:18.200299 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:05:18.200265 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:05:30.062899 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:05:30.062861 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:05:35.696216 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:05:35.696184 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:05:35.840609 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:05:35.840577 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:05:36.029353 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:05:36.029258 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:05:43.414856 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:05:43.414820 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:05:43.558334 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:05:43.558300 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:05:43.675200 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:05:43.675114 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:05:43.824425 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:05:43.824388 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:05:51.293203 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:05:51.293159 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:05:51.457119 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:05:51.457072 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:06:02.870187 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:06:02.870113 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 17:06:02.872163 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:06:02.872138 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 17:06:09.316272 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:06:09.316240 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:06:20.899517 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:06:20.899483 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:06:29.467910 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:06:29.467871 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:06:38.971878 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:06:38.971843 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:06:45.487061 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:06:45.487016 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:06:48.612591 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:06:48.612560 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:06:58.791337 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:06:58.791303 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:07:09.881852 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:07:09.881823 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:07:18.372438 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:07:18.372405 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:07:41.067985 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:07:41.067906 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:07:54.177989 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:07:54.177953 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:08:19.118772 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:08:19.118737 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:08:32.574449 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:08:32.574416 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:08:48.638347 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:08:48.638308 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:08:58.780653 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:08:58.780575 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:09:08.910306 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:09:08.910274 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:09:18.999135 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:09:18.999089 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:09:28.178762 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:09:28.178729 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:09:59.955557 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:09:59.955524 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:10:32.295324 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:10:32.295219 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:11:02.894148 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:11:02.894121 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 17:11:02.896795 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:11:02.896774 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 17:11:17.788922 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:11:17.788887 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:11:28.771845 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:11:28.771809 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:11:39.960716 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:11:39.960676 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:11:50.363324 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:11:50.363288 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:12:00.874739 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:12:00.874657 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:12:11.558832 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:12:11.558801 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:12:21.626301 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:12:21.626254 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:12:32.068958 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:12:32.068925 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:12:43.064184 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:12:43.064152 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:12:51.768531 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:12:51.768493 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:13:05.077640 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:13:05.077602 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:13:09.792739 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:13:09.792709 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:13:22.872611 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:13:22.872578 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:13:31.663609 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:13:31.663519 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:13:41.567400 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:13:41.567359 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:13:53.371921 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:13:53.371882 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:14:01.024245 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:14:01.024211 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:14:13.785551 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:14:13.785509 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:14:23.367281 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:14:23.367246 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:14:32.446795 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:14:32.446765 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:14:42.281345 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:14:42.281306 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:14:51.585665 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:14:51.585625 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:15:00.166809 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:00.166725 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737035-4xtn5"] Jul 16 17:15:00.170062 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:00.170044 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737035-4xtn5" Jul 16 17:15:00.174249 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:00.174220 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t6smk\"" Jul 16 17:15:00.179611 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:00.179591 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737035-4xtn5" Jul 16 17:15:00.230903 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:00.230872 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737035-4xtn5"] Jul 16 17:15:00.517572 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:00.517548 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737035-4xtn5"] Jul 16 17:15:00.519797 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:15:00.519773 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2ffe75_5d81_45c2_b59e_e346e733abf1.slice/crio-3a5cb18fb131be8a62009a000a359b3719b182dbffe55f258ba16accd127a052 WatchSource:0}: Error finding container 3a5cb18fb131be8a62009a000a359b3719b182dbffe55f258ba16accd127a052: Status 404 returned error can't find the container with id 3a5cb18fb131be8a62009a000a359b3719b182dbffe55f258ba16accd127a052 Jul 16 17:15:00.521608 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:00.521592 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:15:00.876560 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:00.876526 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:15:00.896977 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:00.896943 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737035-4xtn5" event={"ID":"3f2ffe75-5d81-45c2-b59e-e346e733abf1","Type":"ContainerStarted","Data":"3a5cb18fb131be8a62009a000a359b3719b182dbffe55f258ba16accd127a052"} Jul 16 17:15:03.913516 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:03.913478 2575 generic.go:356] "Generic (PLEG): container finished" podID="3f2ffe75-5d81-45c2-b59e-e346e733abf1" containerID="6741b43c2773e2001f88a3b12a955190b5c281b62d34e83b98ffcecf5aaed39a" exitCode=0 Jul 16 17:15:03.913962 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:03.913535 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737035-4xtn5" event={"ID":"3f2ffe75-5d81-45c2-b59e-e346e733abf1","Type":"ContainerDied","Data":"6741b43c2773e2001f88a3b12a955190b5c281b62d34e83b98ffcecf5aaed39a"} Jul 16 17:15:05.047399 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:05.047375 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737035-4xtn5" Jul 16 17:15:05.922753 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:05.922723 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737035-4xtn5" Jul 16 17:15:05.922753 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:05.922739 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737035-4xtn5" event={"ID":"3f2ffe75-5d81-45c2-b59e-e346e733abf1","Type":"ContainerDied","Data":"3a5cb18fb131be8a62009a000a359b3719b182dbffe55f258ba16accd127a052"} Jul 16 17:15:05.922968 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:05.922791 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a5cb18fb131be8a62009a000a359b3719b182dbffe55f258ba16accd127a052" Jul 16 17:15:14.786883 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:14.786843 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:15:23.797465 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:23.797431 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:15:33.190172 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:33.190128 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:15:45.286020 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:45.285986 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:15:56.385601 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:15:56.385567 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:16:02.920533 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:16:02.920508 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 17:16:02.924203 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:16:02.924170 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 17:16:06.521204 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:16:06.521165 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:16:22.107272 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:16:22.107238 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:16:29.609408 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:16:29.609331 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:16:34.776789 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:16:34.776752 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:16:44.487182 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:16:44.487150 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:16:55.771583 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:16:55.771543 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:16:57.627812 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:16:57.627774 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:05.437241 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:05.437208 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:07.705016 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:07.704976 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:17:13.713455 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:13.713420 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:18.346926 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:18.346893 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:17:18.964423 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:18.964391 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:22.170889 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:22.170858 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:22.253984 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:22.253947 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:22.712989 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:22.712960 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:27.961793 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:27.961757 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:17:30.015752 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.015719 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf"] Jul 16 17:17:30.016141 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.016076 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f2ffe75-5d81-45c2-b59e-e346e733abf1" containerName="cleanup" Jul 16 17:17:30.016141 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.016089 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f2ffe75-5d81-45c2-b59e-e346e733abf1" containerName="cleanup" Jul 16 17:17:30.016234 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.016169 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f2ffe75-5d81-45c2-b59e-e346e733abf1" containerName="cleanup" Jul 16 17:17:30.019174 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.019154 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" Jul 16 17:17:30.022249 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.022226 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 17:17:30.022371 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.022272 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 17:17:30.023319 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.023300 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bff62\"" Jul 16 17:17:30.023420 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.023299 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ec0988f9-serving-cert\"" Jul 16 17:17:30.032220 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.032197 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf"] Jul 16 17:17:30.157106 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.157060 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm4nh\" (UniqueName: \"kubernetes.io/projected/522954cf-75d3-44de-8bbc-569c9b19743b-kube-api-access-jm4nh\") pod \"maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf\" (UID: \"522954cf-75d3-44de-8bbc-569c9b19743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" Jul 16 17:17:30.157285 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.157185 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/522954cf-75d3-44de-8bbc-569c9b19743b-maas-api-tls\") pod \"maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf\" (UID: \"522954cf-75d3-44de-8bbc-569c9b19743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" Jul 16 17:17:30.157285 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.157242 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/522954cf-75d3-44de-8bbc-569c9b19743b-openshift-service-ca\") pod \"maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf\" (UID: \"522954cf-75d3-44de-8bbc-569c9b19743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" Jul 16 17:17:30.258277 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.258250 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm4nh\" (UniqueName: \"kubernetes.io/projected/522954cf-75d3-44de-8bbc-569c9b19743b-kube-api-access-jm4nh\") pod \"maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf\" (UID: \"522954cf-75d3-44de-8bbc-569c9b19743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" Jul 16 17:17:30.258464 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.258323 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/522954cf-75d3-44de-8bbc-569c9b19743b-maas-api-tls\") pod \"maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf\" (UID: \"522954cf-75d3-44de-8bbc-569c9b19743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" Jul 16 17:17:30.258534 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.258477 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/522954cf-75d3-44de-8bbc-569c9b19743b-openshift-service-ca\") pod \"maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf\" (UID: \"522954cf-75d3-44de-8bbc-569c9b19743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" Jul 16 17:17:30.259069 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.259050 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/522954cf-75d3-44de-8bbc-569c9b19743b-openshift-service-ca\") pod \"maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf\" (UID: \"522954cf-75d3-44de-8bbc-569c9b19743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" Jul 16 17:17:30.260768 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.260749 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/522954cf-75d3-44de-8bbc-569c9b19743b-maas-api-tls\") pod \"maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf\" (UID: \"522954cf-75d3-44de-8bbc-569c9b19743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" Jul 16 17:17:30.268204 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.268180 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm4nh\" (UniqueName: \"kubernetes.io/projected/522954cf-75d3-44de-8bbc-569c9b19743b-kube-api-access-jm4nh\") pod \"maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf\" (UID: \"522954cf-75d3-44de-8bbc-569c9b19743b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" Jul 16 17:17:30.329276 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.329250 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" Jul 16 17:17:30.460528 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.460466 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf"] Jul 16 17:17:30.464231 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:17:30.464202 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522954cf_75d3_44de_8bbc_569c9b19743b.slice/crio-81f3cf8f5e32ebbfb0719e0125772d345e0fd52532deb99b6501cfd8d2fb9ed4 WatchSource:0}: Error finding container 81f3cf8f5e32ebbfb0719e0125772d345e0fd52532deb99b6501cfd8d2fb9ed4: Status 404 returned error can't find the container with id 81f3cf8f5e32ebbfb0719e0125772d345e0fd52532deb99b6501cfd8d2fb9ed4 Jul 16 17:17:30.555787 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:30.555748 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" event={"ID":"522954cf-75d3-44de-8bbc-569c9b19743b","Type":"ContainerStarted","Data":"81f3cf8f5e32ebbfb0719e0125772d345e0fd52532deb99b6501cfd8d2fb9ed4"} Jul 16 17:17:31.467260 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:31.467225 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:31.669270 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:31.669187 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:33.571506 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:33.571466 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" event={"ID":"522954cf-75d3-44de-8bbc-569c9b19743b","Type":"ContainerStarted","Data":"014da8884ec2236fab68571f2192651df485f79037b0f48d0d813a692d66dd27"} Jul 16 17:17:33.571889 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:33.571647 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" Jul 16 17:17:33.592333 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:33.592272 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" podStartSLOduration=2.285035848 podStartE2EDuration="4.592257671s" totalImagesPullingTime="2.307221823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:17:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:17:33.591799025 +0000 UTC m=+1591.110524255" watchObservedRunningTime="2026-07-16 17:17:33.592257671 +0000 UTC m=+1591.110982906" Jul 16 17:17:33.772864 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:33.772840 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:36.323152 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:36.323119 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:36.439157 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:36.439122 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:38.457486 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:38.457459 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:39.581587 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:39.581560 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" Jul 16 17:17:42.822847 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:42.822818 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf"] Jul 16 17:17:42.823260 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:42.823013 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" podUID="522954cf-75d3-44de-8bbc-569c9b19743b" containerName="maas-api" containerID="cri-o://014da8884ec2236fab68571f2192651df485f79037b0f48d0d813a692d66dd27" gracePeriod=30 Jul 16 17:17:42.863518 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:42.863489 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:43.071014 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.070986 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" Jul 16 17:17:43.173823 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.173794 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/522954cf-75d3-44de-8bbc-569c9b19743b-openshift-service-ca\") pod \"522954cf-75d3-44de-8bbc-569c9b19743b\" (UID: \"522954cf-75d3-44de-8bbc-569c9b19743b\") " Jul 16 17:17:43.174020 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.173887 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jm4nh\" (UniqueName: \"kubernetes.io/projected/522954cf-75d3-44de-8bbc-569c9b19743b-kube-api-access-jm4nh\") pod \"522954cf-75d3-44de-8bbc-569c9b19743b\" (UID: \"522954cf-75d3-44de-8bbc-569c9b19743b\") " Jul 16 17:17:43.174020 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.173915 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/522954cf-75d3-44de-8bbc-569c9b19743b-maas-api-tls\") pod \"522954cf-75d3-44de-8bbc-569c9b19743b\" (UID: \"522954cf-75d3-44de-8bbc-569c9b19743b\") " Jul 16 17:17:43.174196 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.174173 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522954cf-75d3-44de-8bbc-569c9b19743b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "522954cf-75d3-44de-8bbc-569c9b19743b" (UID: "522954cf-75d3-44de-8bbc-569c9b19743b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:17:43.176144 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.176110 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522954cf-75d3-44de-8bbc-569c9b19743b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "522954cf-75d3-44de-8bbc-569c9b19743b" (UID: "522954cf-75d3-44de-8bbc-569c9b19743b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:17:43.176262 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.176139 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522954cf-75d3-44de-8bbc-569c9b19743b-kube-api-access-jm4nh" (OuterVolumeSpecName: "kube-api-access-jm4nh") pod "522954cf-75d3-44de-8bbc-569c9b19743b" (UID: "522954cf-75d3-44de-8bbc-569c9b19743b"). InnerVolumeSpecName "kube-api-access-jm4nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:17:43.274996 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.274960 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/522954cf-75d3-44de-8bbc-569c9b19743b-openshift-service-ca\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:17:43.274996 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.274992 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm4nh\" (UniqueName: \"kubernetes.io/projected/522954cf-75d3-44de-8bbc-569c9b19743b-kube-api-access-jm4nh\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:17:43.274996 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.275002 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/522954cf-75d3-44de-8bbc-569c9b19743b-maas-api-tls\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:17:43.614309 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.614280 2575 generic.go:356] "Generic (PLEG): container finished" podID="522954cf-75d3-44de-8bbc-569c9b19743b" containerID="014da8884ec2236fab68571f2192651df485f79037b0f48d0d813a692d66dd27" exitCode=0 Jul 16 17:17:43.614500 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.614320 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" Jul 16 17:17:43.614500 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.614351 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" event={"ID":"522954cf-75d3-44de-8bbc-569c9b19743b","Type":"ContainerDied","Data":"014da8884ec2236fab68571f2192651df485f79037b0f48d0d813a692d66dd27"} Jul 16 17:17:43.614500 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.614402 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf" event={"ID":"522954cf-75d3-44de-8bbc-569c9b19743b","Type":"ContainerDied","Data":"81f3cf8f5e32ebbfb0719e0125772d345e0fd52532deb99b6501cfd8d2fb9ed4"} Jul 16 17:17:43.614500 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.614432 2575 scope.go:117] "RemoveContainer" containerID="014da8884ec2236fab68571f2192651df485f79037b0f48d0d813a692d66dd27" Jul 16 17:17:43.623232 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.623211 2575 scope.go:117] "RemoveContainer" containerID="014da8884ec2236fab68571f2192651df485f79037b0f48d0d813a692d66dd27" Jul 16 17:17:43.623480 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:17:43.623462 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"014da8884ec2236fab68571f2192651df485f79037b0f48d0d813a692d66dd27\": container with ID starting with 014da8884ec2236fab68571f2192651df485f79037b0f48d0d813a692d66dd27 not found: ID does not exist" containerID="014da8884ec2236fab68571f2192651df485f79037b0f48d0d813a692d66dd27" Jul 16 17:17:43.623522 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.623487 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014da8884ec2236fab68571f2192651df485f79037b0f48d0d813a692d66dd27"} err="failed to get container status \"014da8884ec2236fab68571f2192651df485f79037b0f48d0d813a692d66dd27\": rpc error: code = NotFound desc = could not find container \"014da8884ec2236fab68571f2192651df485f79037b0f48d0d813a692d66dd27\": container with ID starting with 014da8884ec2236fab68571f2192651df485f79037b0f48d0d813a692d66dd27 not found: ID does not exist" Jul 16 17:17:43.631167 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.631149 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:43.638059 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.638033 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf"] Jul 16 17:17:43.643189 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:43.643170 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ec0988f9-67b7f6fc8c-4grsf"] Jul 16 17:17:44.909603 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:44.909568 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="522954cf-75d3-44de-8bbc-569c9b19743b" path="/var/lib/kubelet/pods/522954cf-75d3-44de-8bbc-569c9b19743b/volumes" Jul 16 17:17:50.388111 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:50.388073 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:50.520740 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:50.520704 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:50.609314 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:50.609284 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:50.713286 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:50.713252 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:50.807818 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:50.807792 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:51.045069 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.044997 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:51.116364 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.116332 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8"] Jul 16 17:17:51.116874 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.116857 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522954cf-75d3-44de-8bbc-569c9b19743b" containerName="maas-api" Jul 16 17:17:51.116953 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.116878 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="522954cf-75d3-44de-8bbc-569c9b19743b" containerName="maas-api" Jul 16 17:17:51.117006 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.116958 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="522954cf-75d3-44de-8bbc-569c9b19743b" containerName="maas-api" Jul 16 17:17:51.121809 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.121789 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" Jul 16 17:17:51.124497 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.124470 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b593195d-serving-cert\"" Jul 16 17:17:51.124605 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.124510 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 17:17:51.125495 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.125478 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bff62\"" Jul 16 17:17:51.125598 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.125506 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 17:17:51.130041 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.130023 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8"] Jul 16 17:17:51.134962 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.134941 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae48e70-2740-44d8-9aff-ec32f08683bb-openshift-service-ca\") pod \"maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8\" (UID: \"3ae48e70-2740-44d8-9aff-ec32f08683bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" Jul 16 17:17:51.135043 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.135000 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6qxv\" (UniqueName: \"kubernetes.io/projected/3ae48e70-2740-44d8-9aff-ec32f08683bb-kube-api-access-b6qxv\") pod \"maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8\" (UID: \"3ae48e70-2740-44d8-9aff-ec32f08683bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" Jul 16 17:17:51.135043 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.135037 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ae48e70-2740-44d8-9aff-ec32f08683bb-maas-api-tls\") pod \"maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8\" (UID: \"3ae48e70-2740-44d8-9aff-ec32f08683bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" Jul 16 17:17:51.154847 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.154827 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:51.235443 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.235415 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ae48e70-2740-44d8-9aff-ec32f08683bb-maas-api-tls\") pod \"maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8\" (UID: \"3ae48e70-2740-44d8-9aff-ec32f08683bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" Jul 16 17:17:51.235619 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.235472 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae48e70-2740-44d8-9aff-ec32f08683bb-openshift-service-ca\") pod \"maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8\" (UID: \"3ae48e70-2740-44d8-9aff-ec32f08683bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" Jul 16 17:17:51.235619 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.235528 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6qxv\" (UniqueName: \"kubernetes.io/projected/3ae48e70-2740-44d8-9aff-ec32f08683bb-kube-api-access-b6qxv\") pod \"maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8\" (UID: \"3ae48e70-2740-44d8-9aff-ec32f08683bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" Jul 16 17:17:51.235619 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:17:51.235570 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-serving-cert: secret "maas-api-e2e-migrate-b593195d-serving-cert" not found Jul 16 17:17:51.235795 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:17:51.235650 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ae48e70-2740-44d8-9aff-ec32f08683bb-maas-api-tls podName:3ae48e70-2740-44d8-9aff-ec32f08683bb nodeName:}" failed. No retries permitted until 2026-07-16 17:17:51.735632382 +0000 UTC m=+1609.254357595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3ae48e70-2740-44d8-9aff-ec32f08683bb-maas-api-tls") pod "maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" (UID: "3ae48e70-2740-44d8-9aff-ec32f08683bb") : secret "maas-api-e2e-migrate-b593195d-serving-cert" not found Jul 16 17:17:51.236246 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.236219 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae48e70-2740-44d8-9aff-ec32f08683bb-openshift-service-ca\") pod \"maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8\" (UID: \"3ae48e70-2740-44d8-9aff-ec32f08683bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" Jul 16 17:17:51.245913 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.245893 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6qxv\" (UniqueName: \"kubernetes.io/projected/3ae48e70-2740-44d8-9aff-ec32f08683bb-kube-api-access-b6qxv\") pod \"maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8\" (UID: \"3ae48e70-2740-44d8-9aff-ec32f08683bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" Jul 16 17:17:51.734548 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.734512 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:51.738699 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.738680 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ae48e70-2740-44d8-9aff-ec32f08683bb-maas-api-tls\") pod \"maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8\" (UID: \"3ae48e70-2740-44d8-9aff-ec32f08683bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" Jul 16 17:17:51.741054 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.741003 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ae48e70-2740-44d8-9aff-ec32f08683bb-maas-api-tls\") pod \"maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8\" (UID: \"3ae48e70-2740-44d8-9aff-ec32f08683bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" Jul 16 17:17:51.857811 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.857777 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:51.983899 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:51.983862 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:52.033613 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:52.033528 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" Jul 16 17:17:52.198539 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:52.198515 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8"] Jul 16 17:17:52.200885 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:17:52.200858 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae48e70_2740_44d8_9aff_ec32f08683bb.slice/crio-4aea61f84059cee2dad49f66f0521b3a46d124a9b2696fa5f6502d1956501e1e WatchSource:0}: Error finding container 4aea61f84059cee2dad49f66f0521b3a46d124a9b2696fa5f6502d1956501e1e: Status 404 returned error can't find the container with id 4aea61f84059cee2dad49f66f0521b3a46d124a9b2696fa5f6502d1956501e1e Jul 16 17:17:52.661184 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:52.661146 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" event={"ID":"3ae48e70-2740-44d8-9aff-ec32f08683bb","Type":"ContainerStarted","Data":"f7f0523325f1c2d1a1501e39dc786b999f2c5d32ce8ccd85cbd3031fcc0c43dc"} Jul 16 17:17:52.661366 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:52.661234 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" event={"ID":"3ae48e70-2740-44d8-9aff-ec32f08683bb","Type":"ContainerStarted","Data":"4aea61f84059cee2dad49f66f0521b3a46d124a9b2696fa5f6502d1956501e1e"} Jul 16 17:17:52.661366 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:52.661258 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" Jul 16 17:17:52.683964 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:52.683913 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" podStartSLOduration=1.361902109 podStartE2EDuration="1.683897379s" totalImagesPullingTime="321.99527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:17:52.682294336 +0000 UTC m=+1610.201019570" watchObservedRunningTime="2026-07-16 17:17:52.683897379 +0000 UTC m=+1610.202622613" Jul 16 17:17:53.875992 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:53.875958 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:54.472219 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:54.472186 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:17:56.718300 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:56.718266 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:56.814954 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:56.814919 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:58.358807 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:58.358782 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:17:58.669982 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:17:58.669956 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" Jul 16 17:18:01.854557 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:01.854527 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8"] Jul 16 17:18:01.854981 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:01.854743 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" podUID="3ae48e70-2740-44d8-9aff-ec32f08683bb" containerName="maas-api" containerID="cri-o://f7f0523325f1c2d1a1501e39dc786b999f2c5d32ce8ccd85cbd3031fcc0c43dc" gracePeriod=30 Jul 16 17:18:01.884400 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:01.884374 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:02.099649 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.099622 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" Jul 16 17:18:02.116844 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.116785 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae48e70-2740-44d8-9aff-ec32f08683bb-openshift-service-ca\") pod \"3ae48e70-2740-44d8-9aff-ec32f08683bb\" (UID: \"3ae48e70-2740-44d8-9aff-ec32f08683bb\") " Jul 16 17:18:02.116844 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.116834 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6qxv\" (UniqueName: \"kubernetes.io/projected/3ae48e70-2740-44d8-9aff-ec32f08683bb-kube-api-access-b6qxv\") pod \"3ae48e70-2740-44d8-9aff-ec32f08683bb\" (UID: \"3ae48e70-2740-44d8-9aff-ec32f08683bb\") " Jul 16 17:18:02.117020 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.116852 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ae48e70-2740-44d8-9aff-ec32f08683bb-maas-api-tls\") pod \"3ae48e70-2740-44d8-9aff-ec32f08683bb\" (UID: \"3ae48e70-2740-44d8-9aff-ec32f08683bb\") " Jul 16 17:18:02.117151 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.117125 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae48e70-2740-44d8-9aff-ec32f08683bb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3ae48e70-2740-44d8-9aff-ec32f08683bb" (UID: "3ae48e70-2740-44d8-9aff-ec32f08683bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:18:02.118957 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.118936 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae48e70-2740-44d8-9aff-ec32f08683bb-kube-api-access-b6qxv" (OuterVolumeSpecName: "kube-api-access-b6qxv") pod "3ae48e70-2740-44d8-9aff-ec32f08683bb" (UID: "3ae48e70-2740-44d8-9aff-ec32f08683bb"). InnerVolumeSpecName "kube-api-access-b6qxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:18:02.119040 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.118933 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ae48e70-2740-44d8-9aff-ec32f08683bb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3ae48e70-2740-44d8-9aff-ec32f08683bb" (UID: "3ae48e70-2740-44d8-9aff-ec32f08683bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:18:02.218336 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.218298 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae48e70-2740-44d8-9aff-ec32f08683bb-openshift-service-ca\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:18:02.218336 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.218329 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6qxv\" (UniqueName: \"kubernetes.io/projected/3ae48e70-2740-44d8-9aff-ec32f08683bb-kube-api-access-b6qxv\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:18:02.218336 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.218339 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ae48e70-2740-44d8-9aff-ec32f08683bb-maas-api-tls\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:18:02.705338 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.705306 2575 generic.go:356] "Generic (PLEG): container finished" podID="3ae48e70-2740-44d8-9aff-ec32f08683bb" containerID="f7f0523325f1c2d1a1501e39dc786b999f2c5d32ce8ccd85cbd3031fcc0c43dc" exitCode=0 Jul 16 17:18:02.705517 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.705348 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" Jul 16 17:18:02.705517 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.705349 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" event={"ID":"3ae48e70-2740-44d8-9aff-ec32f08683bb","Type":"ContainerDied","Data":"f7f0523325f1c2d1a1501e39dc786b999f2c5d32ce8ccd85cbd3031fcc0c43dc"} Jul 16 17:18:02.705517 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.705411 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8" event={"ID":"3ae48e70-2740-44d8-9aff-ec32f08683bb","Type":"ContainerDied","Data":"4aea61f84059cee2dad49f66f0521b3a46d124a9b2696fa5f6502d1956501e1e"} Jul 16 17:18:02.705517 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.705440 2575 scope.go:117] "RemoveContainer" containerID="f7f0523325f1c2d1a1501e39dc786b999f2c5d32ce8ccd85cbd3031fcc0c43dc" Jul 16 17:18:02.714240 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.714212 2575 scope.go:117] "RemoveContainer" containerID="f7f0523325f1c2d1a1501e39dc786b999f2c5d32ce8ccd85cbd3031fcc0c43dc" Jul 16 17:18:02.714487 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:18:02.714470 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f0523325f1c2d1a1501e39dc786b999f2c5d32ce8ccd85cbd3031fcc0c43dc\": container with ID starting with f7f0523325f1c2d1a1501e39dc786b999f2c5d32ce8ccd85cbd3031fcc0c43dc not found: ID does not exist" containerID="f7f0523325f1c2d1a1501e39dc786b999f2c5d32ce8ccd85cbd3031fcc0c43dc" Jul 16 17:18:02.714531 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.714497 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f0523325f1c2d1a1501e39dc786b999f2c5d32ce8ccd85cbd3031fcc0c43dc"} err="failed to get container status \"f7f0523325f1c2d1a1501e39dc786b999f2c5d32ce8ccd85cbd3031fcc0c43dc\": rpc error: code = NotFound desc = could not find container \"f7f0523325f1c2d1a1501e39dc786b999f2c5d32ce8ccd85cbd3031fcc0c43dc\": container with ID starting with f7f0523325f1c2d1a1501e39dc786b999f2c5d32ce8ccd85cbd3031fcc0c43dc not found: ID does not exist" Jul 16 17:18:02.727574 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.727549 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8"] Jul 16 17:18:02.732203 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.732183 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b593195d-5577f9f46b-lnns8"] Jul 16 17:18:02.909913 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:02.909846 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ae48e70-2740-44d8-9aff-ec32f08683bb" path="/var/lib/kubelet/pods/3ae48e70-2740-44d8-9aff-ec32f08683bb/volumes" Jul 16 17:18:03.980863 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:03.980829 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:05.873351 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:05.873315 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:18:10.461012 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:10.460980 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:10.573982 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:10.573942 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:10.785601 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:10.785557 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:10.884477 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:10.884434 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:10.967192 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:10.967158 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s"] Jul 16 17:18:10.967635 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:10.967620 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ae48e70-2740-44d8-9aff-ec32f08683bb" containerName="maas-api" Jul 16 17:18:10.967678 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:10.967639 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ae48e70-2740-44d8-9aff-ec32f08683bb" containerName="maas-api" Jul 16 17:18:10.967741 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:10.967730 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ae48e70-2740-44d8-9aff-ec32f08683bb" containerName="maas-api" Jul 16 17:18:10.974249 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:10.974225 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" Jul 16 17:18:10.978304 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:10.978285 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bff62\"" Jul 16 17:18:10.979321 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:10.979303 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c527f463-serving-cert\"" Jul 16 17:18:10.979451 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:10.979341 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 17:18:10.979451 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:10.979369 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 17:18:10.987456 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:10.987426 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s"] Jul 16 17:18:10.989616 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:10.989595 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:11.085666 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:11.085583 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:11.094729 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:11.094694 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-maas-api-tls\") pod \"maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s\" (UID: \"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" Jul 16 17:18:11.094890 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:11.094757 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-openshift-service-ca\") pod \"maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s\" (UID: \"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" Jul 16 17:18:11.094890 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:11.094857 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftpxw\" (UniqueName: \"kubernetes.io/projected/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-kube-api-access-ftpxw\") pod \"maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s\" (UID: \"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" Jul 16 17:18:11.195796 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:11.195758 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-openshift-service-ca\") pod \"maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s\" (UID: \"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" Jul 16 17:18:11.195983 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:11.195838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftpxw\" (UniqueName: \"kubernetes.io/projected/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-kube-api-access-ftpxw\") pod \"maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s\" (UID: \"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" Jul 16 17:18:11.195983 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:11.195906 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-maas-api-tls\") pod \"maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s\" (UID: \"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" Jul 16 17:18:11.196118 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:18:11.196023 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-serving-cert: secret "maas-api-e2e-derive-c527f463-serving-cert" not found Jul 16 17:18:11.196118 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:18:11.196089 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-maas-api-tls podName:db870ed3-2bdb-4c7c-a5ff-7d2010b020fd nodeName:}" failed. No retries permitted until 2026-07-16 17:18:11.696069288 +0000 UTC m=+1629.214794515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-maas-api-tls") pod "maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" (UID: "db870ed3-2bdb-4c7c-a5ff-7d2010b020fd") : secret "maas-api-e2e-derive-c527f463-serving-cert" not found Jul 16 17:18:11.196391 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:11.196371 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-openshift-service-ca\") pod \"maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s\" (UID: \"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" Jul 16 17:18:11.209936 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:11.209908 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftpxw\" (UniqueName: \"kubernetes.io/projected/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-kube-api-access-ftpxw\") pod \"maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s\" (UID: \"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" Jul 16 17:18:11.598744 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:11.598708 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:11.699709 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:11.699674 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-maas-api-tls\") pod \"maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s\" (UID: \"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" Jul 16 17:18:11.699908 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:18:11.699825 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-serving-cert: secret "maas-api-e2e-derive-c527f463-serving-cert" not found Jul 16 17:18:11.699908 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:18:11.699902 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-maas-api-tls podName:db870ed3-2bdb-4c7c-a5ff-7d2010b020fd nodeName:}" failed. No retries permitted until 2026-07-16 17:18:12.699887228 +0000 UTC m=+1630.218612444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-maas-api-tls") pod "maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" (UID: "db870ed3-2bdb-4c7c-a5ff-7d2010b020fd") : secret "maas-api-e2e-derive-c527f463-serving-cert" not found Jul 16 17:18:11.975051 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:11.975016 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:12.077911 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:12.077882 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:12.709322 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:12.709284 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-maas-api-tls\") pod \"maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s\" (UID: \"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" Jul 16 17:18:12.711741 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:12.711720 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-maas-api-tls\") pod \"maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s\" (UID: \"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" Jul 16 17:18:12.784893 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:12.784859 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" Jul 16 17:18:12.956669 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:12.956637 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s"] Jul 16 17:18:12.960731 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:18:12.960673 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb870ed3_2bdb_4c7c_a5ff_7d2010b020fd.slice/crio-a78e57c5c04db8227bf948f882758b3393abfbe2fc5981da82bf6cb540971914 WatchSource:0}: Error finding container a78e57c5c04db8227bf948f882758b3393abfbe2fc5981da82bf6cb540971914: Status 404 returned error can't find the container with id a78e57c5c04db8227bf948f882758b3393abfbe2fc5981da82bf6cb540971914 Jul 16 17:18:13.757385 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:13.757347 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" event={"ID":"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd","Type":"ContainerStarted","Data":"a78e57c5c04db8227bf948f882758b3393abfbe2fc5981da82bf6cb540971914"} Jul 16 17:18:14.127584 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:14.127554 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:14.567305 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:14.567267 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:18:15.767231 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:15.767195 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" event={"ID":"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd","Type":"ContainerStarted","Data":"50c06c2285ab89c0307c52ca4ebcb2bcf2eb7286450b1a042f9192de155883c7"} Jul 16 17:18:15.767639 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:15.767310 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" Jul 16 17:18:16.494697 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:16.494664 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:16.596314 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:16.596279 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:18.522076 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:18.522048 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:21.779293 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:21.779266 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" Jul 16 17:18:21.801863 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:21.801814 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" podStartSLOduration=9.477708185000001 podStartE2EDuration="11.801799686s" totalImagesPullingTime="2.324091501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:18:15.795667547 +0000 UTC m=+1633.314392781" watchObservedRunningTime="2026-07-16 17:18:21.801799686 +0000 UTC m=+1639.320524919" Jul 16 17:18:21.995842 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:21.995809 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s"] Jul 16 17:18:21.996003 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:21.995972 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" podUID="db870ed3-2bdb-4c7c-a5ff-7d2010b020fd" containerName="maas-api" containerID="cri-o://50c06c2285ab89c0307c52ca4ebcb2bcf2eb7286450b1a042f9192de155883c7" gracePeriod=30 Jul 16 17:18:22.110077 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.110054 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:22.245391 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.245369 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" Jul 16 17:18:22.295928 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.295904 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftpxw\" (UniqueName: \"kubernetes.io/projected/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-kube-api-access-ftpxw\") pod \"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd\" (UID: \"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd\") " Jul 16 17:18:22.296060 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.295984 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-openshift-service-ca\") pod \"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd\" (UID: \"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd\") " Jul 16 17:18:22.296060 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.296034 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-maas-api-tls\") pod \"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd\" (UID: \"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd\") " Jul 16 17:18:22.296388 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.296357 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "db870ed3-2bdb-4c7c-a5ff-7d2010b020fd" (UID: "db870ed3-2bdb-4c7c-a5ff-7d2010b020fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:18:22.298003 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.297981 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-kube-api-access-ftpxw" (OuterVolumeSpecName: "kube-api-access-ftpxw") pod "db870ed3-2bdb-4c7c-a5ff-7d2010b020fd" (UID: "db870ed3-2bdb-4c7c-a5ff-7d2010b020fd"). InnerVolumeSpecName "kube-api-access-ftpxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:18:22.298003 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.297994 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "db870ed3-2bdb-4c7c-a5ff-7d2010b020fd" (UID: "db870ed3-2bdb-4c7c-a5ff-7d2010b020fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:18:22.397241 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.397185 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-openshift-service-ca\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:18:22.397241 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.397207 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-maas-api-tls\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:18:22.397241 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.397216 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftpxw\" (UniqueName: \"kubernetes.io/projected/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd-kube-api-access-ftpxw\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:18:22.799975 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.799892 2575 generic.go:356] "Generic (PLEG): container finished" podID="db870ed3-2bdb-4c7c-a5ff-7d2010b020fd" containerID="50c06c2285ab89c0307c52ca4ebcb2bcf2eb7286450b1a042f9192de155883c7" exitCode=0 Jul 16 17:18:22.799975 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.799943 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" event={"ID":"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd","Type":"ContainerDied","Data":"50c06c2285ab89c0307c52ca4ebcb2bcf2eb7286450b1a042f9192de155883c7"} Jul 16 17:18:22.799975 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.799944 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" Jul 16 17:18:22.800564 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.799982 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s" event={"ID":"db870ed3-2bdb-4c7c-a5ff-7d2010b020fd","Type":"ContainerDied","Data":"a78e57c5c04db8227bf948f882758b3393abfbe2fc5981da82bf6cb540971914"} Jul 16 17:18:22.800564 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.800003 2575 scope.go:117] "RemoveContainer" containerID="50c06c2285ab89c0307c52ca4ebcb2bcf2eb7286450b1a042f9192de155883c7" Jul 16 17:18:22.808774 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.808755 2575 scope.go:117] "RemoveContainer" containerID="50c06c2285ab89c0307c52ca4ebcb2bcf2eb7286450b1a042f9192de155883c7" Jul 16 17:18:22.809045 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:18:22.809027 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50c06c2285ab89c0307c52ca4ebcb2bcf2eb7286450b1a042f9192de155883c7\": container with ID starting with 50c06c2285ab89c0307c52ca4ebcb2bcf2eb7286450b1a042f9192de155883c7 not found: ID does not exist" containerID="50c06c2285ab89c0307c52ca4ebcb2bcf2eb7286450b1a042f9192de155883c7" Jul 16 17:18:22.809110 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.809053 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50c06c2285ab89c0307c52ca4ebcb2bcf2eb7286450b1a042f9192de155883c7"} err="failed to get container status \"50c06c2285ab89c0307c52ca4ebcb2bcf2eb7286450b1a042f9192de155883c7\": rpc error: code = NotFound desc = could not find container \"50c06c2285ab89c0307c52ca4ebcb2bcf2eb7286450b1a042f9192de155883c7\": container with ID starting with 50c06c2285ab89c0307c52ca4ebcb2bcf2eb7286450b1a042f9192de155883c7 not found: ID does not exist" Jul 16 17:18:22.824345 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.824320 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s"] Jul 16 17:18:22.829249 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.829221 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c527f463-77cb797b5b-7fn2s"] Jul 16 17:18:22.911668 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:22.911630 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db870ed3-2bdb-4c7c-a5ff-7d2010b020fd" path="/var/lib/kubelet/pods/db870ed3-2bdb-4c7c-a5ff-7d2010b020fd/volumes" Jul 16 17:18:23.124045 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:23.124014 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:25.978499 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:25.978462 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:18:29.906396 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:29.906365 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:30.035852 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:30.035816 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:30.438331 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:30.438296 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:30.530827 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:30.530797 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:30.630305 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:30.630277 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:32.144612 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:32.144583 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:39.050366 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:39.050321 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:39.161257 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:39.161222 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:39.262312 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:39.262280 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:39.375930 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:39.375906 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:40.014794 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:40.014767 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:18:42.799852 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:18:42.799828 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:19:05.994247 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:19:05.994196 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:19:12.830463 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:19:12.830430 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:19:13.003371 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:19:13.003339 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:19:13.904577 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:19:13.904547 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:19:29.218447 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:19:29.218365 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:19:37.903149 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:19:37.903117 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:19:38.035830 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:19:38.035796 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:19:38.162841 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:19:38.162756 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:19:48.796422 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:19:48.796390 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:19:49.668297 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:19:49.668264 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:19:57.559676 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:19:57.559642 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:19:58.542334 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:19:58.542304 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:19:59.682598 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:19:59.682564 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:19:59.889936 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:19:59.889903 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:01.025706 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:01.025674 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:01.403553 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:01.403521 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:20:09.310378 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:09.310343 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:09.461661 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:09.461633 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:09.634208 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:09.634176 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:20:10.100255 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:10.100160 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:16.707492 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:16.707459 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:20:18.533799 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:18.533766 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:18.648994 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:18.648962 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:20.167306 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:20.167266 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:20.280634 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:20.280595 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:20.502305 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:20.502223 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:21.006304 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:21.006270 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:21.308882 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:21.308801 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:24.031123 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.031067 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:24.157922 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.157893 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6"] Jul 16 17:20:24.158308 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.158293 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db870ed3-2bdb-4c7c-a5ff-7d2010b020fd" containerName="maas-api" Jul 16 17:20:24.158372 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.158310 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="db870ed3-2bdb-4c7c-a5ff-7d2010b020fd" containerName="maas-api" Jul 16 17:20:24.158372 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.158367 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="db870ed3-2bdb-4c7c-a5ff-7d2010b020fd" containerName="maas-api" Jul 16 17:20:24.161328 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.161304 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" Jul 16 17:20:24.169849 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.169820 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0c09a637-serving-cert\"" Jul 16 17:20:24.169966 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.169833 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 17:20:24.169966 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.169827 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bff62\"" Jul 16 17:20:24.170192 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.170176 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 17:20:24.199137 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.199109 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6"] Jul 16 17:20:24.202693 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.202666 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:24.220813 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.220791 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3635a036-d8f3-4372-b3c4-221192e554be-maas-api-tls\") pod \"maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6\" (UID: \"3635a036-d8f3-4372-b3c4-221192e554be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" Jul 16 17:20:24.220936 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.220839 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sblh6\" (UniqueName: \"kubernetes.io/projected/3635a036-d8f3-4372-b3c4-221192e554be-kube-api-access-sblh6\") pod \"maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6\" (UID: \"3635a036-d8f3-4372-b3c4-221192e554be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" Jul 16 17:20:24.220936 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.220860 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3635a036-d8f3-4372-b3c4-221192e554be-openshift-service-ca\") pod \"maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6\" (UID: \"3635a036-d8f3-4372-b3c4-221192e554be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" Jul 16 17:20:24.318836 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.318758 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:24.321968 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.321941 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sblh6\" (UniqueName: \"kubernetes.io/projected/3635a036-d8f3-4372-b3c4-221192e554be-kube-api-access-sblh6\") pod \"maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6\" (UID: \"3635a036-d8f3-4372-b3c4-221192e554be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" Jul 16 17:20:24.322141 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.321972 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3635a036-d8f3-4372-b3c4-221192e554be-openshift-service-ca\") pod \"maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6\" (UID: \"3635a036-d8f3-4372-b3c4-221192e554be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" Jul 16 17:20:24.322141 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.322036 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3635a036-d8f3-4372-b3c4-221192e554be-maas-api-tls\") pod \"maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6\" (UID: \"3635a036-d8f3-4372-b3c4-221192e554be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" Jul 16 17:20:24.322266 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:20:24.322201 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-serving-cert: secret "maas-api-e2e-mt-0c09a637-serving-cert" not found Jul 16 17:20:24.322320 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:20:24.322270 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3635a036-d8f3-4372-b3c4-221192e554be-maas-api-tls podName:3635a036-d8f3-4372-b3c4-221192e554be nodeName:}" failed. No retries permitted until 2026-07-16 17:20:24.822250563 +0000 UTC m=+1762.340975775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3635a036-d8f3-4372-b3c4-221192e554be-maas-api-tls") pod "maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" (UID: "3635a036-d8f3-4372-b3c4-221192e554be") : secret "maas-api-e2e-mt-0c09a637-serving-cert" not found Jul 16 17:20:24.322742 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.322723 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3635a036-d8f3-4372-b3c4-221192e554be-openshift-service-ca\") pod \"maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6\" (UID: \"3635a036-d8f3-4372-b3c4-221192e554be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" Jul 16 17:20:24.342966 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.342939 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sblh6\" (UniqueName: \"kubernetes.io/projected/3635a036-d8f3-4372-b3c4-221192e554be-kube-api-access-sblh6\") pod \"maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6\" (UID: \"3635a036-d8f3-4372-b3c4-221192e554be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" Jul 16 17:20:24.566342 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.566308 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:24.826649 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.826616 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3635a036-d8f3-4372-b3c4-221192e554be-maas-api-tls\") pod \"maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6\" (UID: \"3635a036-d8f3-4372-b3c4-221192e554be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" Jul 16 17:20:24.829053 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:24.829025 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3635a036-d8f3-4372-b3c4-221192e554be-maas-api-tls\") pod \"maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6\" (UID: \"3635a036-d8f3-4372-b3c4-221192e554be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" Jul 16 17:20:25.071774 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:25.071742 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" Jul 16 17:20:25.130524 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:25.130470 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:25.239268 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:25.239240 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6"] Jul 16 17:20:25.240865 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:20:25.240837 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3635a036_d8f3_4372_b3c4_221192e554be.slice/crio-9f11fb85278c128ba7983f6ab2ff6a7c11a2d0ecee069b942bb7a848c499fc92 WatchSource:0}: Error finding container 9f11fb85278c128ba7983f6ab2ff6a7c11a2d0ecee069b942bb7a848c499fc92: Status 404 returned error can't find the container with id 9f11fb85278c128ba7983f6ab2ff6a7c11a2d0ecee069b942bb7a848c499fc92 Jul 16 17:20:25.242266 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:25.242244 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:20:25.259169 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:25.259144 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:25.332961 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:25.332879 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" event={"ID":"3635a036-d8f3-4372-b3c4-221192e554be","Type":"ContainerStarted","Data":"9f11fb85278c128ba7983f6ab2ff6a7c11a2d0ecee069b942bb7a848c499fc92"} Jul 16 17:20:28.346902 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:28.346865 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" event={"ID":"3635a036-d8f3-4372-b3c4-221192e554be","Type":"ContainerStarted","Data":"e34a597fd8a0e2b599feff56212df8983d4a22e831d4cc27a1f71ebfaa0eb784"} Jul 16 17:20:28.347304 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:28.346977 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" Jul 16 17:20:28.373128 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:28.373044 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" podStartSLOduration=1.860141941 podStartE2EDuration="4.373027306s" totalImagesPullingTime="2.512885365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:20:28.370457208 +0000 UTC m=+1765.889182442" watchObservedRunningTime="2026-07-16 17:20:28.373027306 +0000 UTC m=+1765.891752539" Jul 16 17:20:29.497443 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:29.497409 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:29.691088 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:29.691052 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:20:30.774075 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:30.774044 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:30.915250 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:30.915221 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:34.357548 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:34.357521 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" Jul 16 17:20:38.718066 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:38.718036 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:38.836464 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:38.836433 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:39.592052 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:39.592001 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:44.673668 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:44.673637 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6"] Jul 16 17:20:44.674189 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:44.673853 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" podUID="3635a036-d8f3-4372-b3c4-221192e554be" containerName="maas-api" containerID="cri-o://e34a597fd8a0e2b599feff56212df8983d4a22e831d4cc27a1f71ebfaa0eb784" gracePeriod=30 Jul 16 17:20:44.712530 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:44.712500 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:44.918825 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:44.918802 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" Jul 16 17:20:45.002482 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:45.002411 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3635a036-d8f3-4372-b3c4-221192e554be-openshift-service-ca\") pod \"3635a036-d8f3-4372-b3c4-221192e554be\" (UID: \"3635a036-d8f3-4372-b3c4-221192e554be\") " Jul 16 17:20:45.002482 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:45.002447 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3635a036-d8f3-4372-b3c4-221192e554be-maas-api-tls\") pod \"3635a036-d8f3-4372-b3c4-221192e554be\" (UID: \"3635a036-d8f3-4372-b3c4-221192e554be\") " Jul 16 17:20:45.002668 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:45.002497 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sblh6\" (UniqueName: \"kubernetes.io/projected/3635a036-d8f3-4372-b3c4-221192e554be-kube-api-access-sblh6\") pod \"3635a036-d8f3-4372-b3c4-221192e554be\" (UID: \"3635a036-d8f3-4372-b3c4-221192e554be\") " Jul 16 17:20:45.002856 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:45.002828 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3635a036-d8f3-4372-b3c4-221192e554be-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3635a036-d8f3-4372-b3c4-221192e554be" (UID: "3635a036-d8f3-4372-b3c4-221192e554be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:20:45.004698 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:45.004675 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3635a036-d8f3-4372-b3c4-221192e554be-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3635a036-d8f3-4372-b3c4-221192e554be" (UID: "3635a036-d8f3-4372-b3c4-221192e554be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:20:45.004806 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:45.004766 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3635a036-d8f3-4372-b3c4-221192e554be-kube-api-access-sblh6" (OuterVolumeSpecName: "kube-api-access-sblh6") pod "3635a036-d8f3-4372-b3c4-221192e554be" (UID: "3635a036-d8f3-4372-b3c4-221192e554be"). InnerVolumeSpecName "kube-api-access-sblh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:20:45.103674 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:45.103643 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3635a036-d8f3-4372-b3c4-221192e554be-openshift-service-ca\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:20:45.103674 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:45.103670 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3635a036-d8f3-4372-b3c4-221192e554be-maas-api-tls\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:20:45.103674 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:45.103679 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sblh6\" (UniqueName: \"kubernetes.io/projected/3635a036-d8f3-4372-b3c4-221192e554be-kube-api-access-sblh6\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:20:45.425328 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:45.425295 2575 generic.go:356] "Generic (PLEG): container finished" podID="3635a036-d8f3-4372-b3c4-221192e554be" containerID="e34a597fd8a0e2b599feff56212df8983d4a22e831d4cc27a1f71ebfaa0eb784" exitCode=0 Jul 16 17:20:45.425499 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:45.425337 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" Jul 16 17:20:45.425499 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:45.425352 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" event={"ID":"3635a036-d8f3-4372-b3c4-221192e554be","Type":"ContainerDied","Data":"e34a597fd8a0e2b599feff56212df8983d4a22e831d4cc27a1f71ebfaa0eb784"} Jul 16 17:20:45.425499 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:45.425393 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6" event={"ID":"3635a036-d8f3-4372-b3c4-221192e554be","Type":"ContainerDied","Data":"9f11fb85278c128ba7983f6ab2ff6a7c11a2d0ecee069b942bb7a848c499fc92"} Jul 16 17:20:45.425499 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:45.425415 2575 scope.go:117] "RemoveContainer" containerID="e34a597fd8a0e2b599feff56212df8983d4a22e831d4cc27a1f71ebfaa0eb784" Jul 16 17:20:45.434554 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:45.434536 2575 scope.go:117] "RemoveContainer" containerID="e34a597fd8a0e2b599feff56212df8983d4a22e831d4cc27a1f71ebfaa0eb784" Jul 16 17:20:45.434812 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:20:45.434795 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34a597fd8a0e2b599feff56212df8983d4a22e831d4cc27a1f71ebfaa0eb784\": container with ID starting with e34a597fd8a0e2b599feff56212df8983d4a22e831d4cc27a1f71ebfaa0eb784 not found: ID does not exist" containerID="e34a597fd8a0e2b599feff56212df8983d4a22e831d4cc27a1f71ebfaa0eb784" Jul 16 17:20:45.434864 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:45.434821 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34a597fd8a0e2b599feff56212df8983d4a22e831d4cc27a1f71ebfaa0eb784"} err="failed to get container status \"e34a597fd8a0e2b599feff56212df8983d4a22e831d4cc27a1f71ebfaa0eb784\": rpc error: code = NotFound desc = could not find container \"e34a597fd8a0e2b599feff56212df8983d4a22e831d4cc27a1f71ebfaa0eb784\": container with ID starting with e34a597fd8a0e2b599feff56212df8983d4a22e831d4cc27a1f71ebfaa0eb784 not found: ID does not exist" Jul 16 17:20:45.452510 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:45.452490 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6"] Jul 16 17:20:45.461264 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:45.461244 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0c09a637-b9b5c77f5-btjw6"] Jul 16 17:20:46.909184 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:46.909154 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3635a036-d8f3-4372-b3c4-221192e554be" path="/var/lib/kubelet/pods/3635a036-d8f3-4372-b3c4-221192e554be/volumes" Jul 16 17:20:47.033055 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:47.033023 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:47.810644 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:47.810611 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:20:54.079798 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:54.079769 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:54.185670 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:54.185639 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:54.366903 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:54.366865 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:54.629270 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:54.629193 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:54.767481 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:54.767448 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:54.900618 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:54.900512 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:20:56.963542 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:20:56.963511 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:21:02.794836 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:02.794806 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:21:02.911078 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:02.911050 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:21:02.950054 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:02.950025 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 17:21:02.953775 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:02.953754 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 17:21:03.027632 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:03.027596 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:21:09.741160 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:09.741130 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:21:09.851768 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:09.851740 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:21:17.814850 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:17.814787 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:21:17.924565 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:17.924533 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:21:18.027517 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:18.027488 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:21:18.860329 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:18.860298 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:21:18.992838 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:18.992803 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:21:20.574616 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:20.574586 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:21:22.587829 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:22.587795 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:21:27.933926 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:27.933893 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:21:28.035485 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:28.035450 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:21:28.662216 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:28.662190 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:21:33.100661 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:33.100625 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:21:36.567118 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:36.567067 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:21:36.753548 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:36.753520 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:21:36.892813 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:36.892787 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:21:47.439974 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:47.439919 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:21:47.868022 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:21:47.867988 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:08.931074 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:08.931043 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:16.573780 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:16.573750 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:16.672244 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:16.672215 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:16.987556 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:16.987527 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:17.586907 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:17.586865 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:17.908796 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:17.908748 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:19.564160 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.564128 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:19.719609 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.719575 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp"] Jul 16 17:22:19.719969 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.719952 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3635a036-d8f3-4372-b3c4-221192e554be" containerName="maas-api" Jul 16 17:22:19.720058 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.719972 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3635a036-d8f3-4372-b3c4-221192e554be" containerName="maas-api" Jul 16 17:22:19.720133 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.720066 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3635a036-d8f3-4372-b3c4-221192e554be" containerName="maas-api" Jul 16 17:22:19.723065 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.723043 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" Jul 16 17:22:19.729912 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.729888 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7b9f8f-serving-cert\"" Jul 16 17:22:19.730037 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.729931 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 17:22:19.730037 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.730010 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 17:22:19.731190 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.731163 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bff62\"" Jul 16 17:22:19.738885 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.738863 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp"] Jul 16 17:22:19.798897 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.798858 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snvmq\" (UniqueName: \"kubernetes.io/projected/cbc3aa04-255b-4957-aa36-d5784fe138df-kube-api-access-snvmq\") pod \"maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp\" (UID: \"cbc3aa04-255b-4957-aa36-d5784fe138df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" Jul 16 17:22:19.799134 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.799073 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc3aa04-255b-4957-aa36-d5784fe138df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp\" (UID: \"cbc3aa04-255b-4957-aa36-d5784fe138df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" Jul 16 17:22:19.799273 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.799152 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc3aa04-255b-4957-aa36-d5784fe138df-maas-api-tls\") pod \"maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp\" (UID: \"cbc3aa04-255b-4957-aa36-d5784fe138df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" Jul 16 17:22:19.875261 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.875227 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:19.900589 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.900559 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc3aa04-255b-4957-aa36-d5784fe138df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp\" (UID: \"cbc3aa04-255b-4957-aa36-d5784fe138df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" Jul 16 17:22:19.900763 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.900601 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc3aa04-255b-4957-aa36-d5784fe138df-maas-api-tls\") pod \"maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp\" (UID: \"cbc3aa04-255b-4957-aa36-d5784fe138df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" Jul 16 17:22:19.900763 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.900629 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snvmq\" (UniqueName: \"kubernetes.io/projected/cbc3aa04-255b-4957-aa36-d5784fe138df-kube-api-access-snvmq\") pod \"maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp\" (UID: \"cbc3aa04-255b-4957-aa36-d5784fe138df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" Jul 16 17:22:19.900763 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:22:19.900719 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-serving-cert: secret "maas-api-e2e-inf-a-7b9f8f-serving-cert" not found Jul 16 17:22:19.900935 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:22:19.900780 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc3aa04-255b-4957-aa36-d5784fe138df-maas-api-tls podName:cbc3aa04-255b-4957-aa36-d5784fe138df nodeName:}" failed. No retries permitted until 2026-07-16 17:22:20.400762769 +0000 UTC m=+1877.919487981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cbc3aa04-255b-4957-aa36-d5784fe138df-maas-api-tls") pod "maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" (UID: "cbc3aa04-255b-4957-aa36-d5784fe138df") : secret "maas-api-e2e-inf-a-7b9f8f-serving-cert" not found Jul 16 17:22:19.901318 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.901294 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc3aa04-255b-4957-aa36-d5784fe138df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp\" (UID: \"cbc3aa04-255b-4957-aa36-d5784fe138df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" Jul 16 17:22:19.912704 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.912680 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snvmq\" (UniqueName: \"kubernetes.io/projected/cbc3aa04-255b-4957-aa36-d5784fe138df-kube-api-access-snvmq\") pod \"maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp\" (UID: \"cbc3aa04-255b-4957-aa36-d5784fe138df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" Jul 16 17:22:19.979610 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:19.979580 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:20.241828 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:20.241754 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:20.405241 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:20.405208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc3aa04-255b-4957-aa36-d5784fe138df-maas-api-tls\") pod \"maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp\" (UID: \"cbc3aa04-255b-4957-aa36-d5784fe138df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" Jul 16 17:22:20.407590 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:20.407564 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc3aa04-255b-4957-aa36-d5784fe138df-maas-api-tls\") pod \"maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp\" (UID: \"cbc3aa04-255b-4957-aa36-d5784fe138df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" Jul 16 17:22:20.491011 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:20.490977 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:20.611525 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:20.611494 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:20.633448 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:20.633417 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" Jul 16 17:22:20.808666 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:20.808640 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp"] Jul 16 17:22:20.811454 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:22:20.811429 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc3aa04_255b_4957_aa36_d5784fe138df.slice/crio-a6e0a90ce1b613245cbf786686eda954459922070a5c9206bb7863eef296f231 WatchSource:0}: Error finding container a6e0a90ce1b613245cbf786686eda954459922070a5c9206bb7863eef296f231: Status 404 returned error can't find the container with id a6e0a90ce1b613245cbf786686eda954459922070a5c9206bb7863eef296f231 Jul 16 17:22:20.841168 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:20.841136 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" event={"ID":"cbc3aa04-255b-4957-aa36-d5784fe138df","Type":"ContainerStarted","Data":"a6e0a90ce1b613245cbf786686eda954459922070a5c9206bb7863eef296f231"} Jul 16 17:22:23.856213 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:23.856174 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" event={"ID":"cbc3aa04-255b-4957-aa36-d5784fe138df","Type":"ContainerStarted","Data":"45e9862b2072b8928271497bfbfc115520235bf97ee0f93bf06755302b5c473a"} Jul 16 17:22:23.856579 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:23.856285 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" Jul 16 17:22:23.883149 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:23.883086 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" podStartSLOduration=2.236236557 podStartE2EDuration="4.883068677s" totalImagesPullingTime="2.64683212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:22:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:22:23.881359034 +0000 UTC m=+1881.400084269" watchObservedRunningTime="2026-07-16 17:22:23.883068677 +0000 UTC m=+1881.401793911" Jul 16 17:22:25.133370 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:25.133288 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:25.500204 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:25.500120 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:25.836005 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:25.835927 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc"] Jul 16 17:22:25.839595 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:25.839574 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:25.844353 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:25.844332 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-901aa3-openshift-default-dockercfg-5wcq5\"" Jul 16 17:22:25.854185 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:25.854156 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc"] Jul 16 17:22:25.873548 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:25.873528 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:25.950778 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:25.950746 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-podinfo\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:25.951082 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:25.951062 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-data\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:25.951270 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:25.951254 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21eb266b-7a0d-43d6-b601-365ad87bec7b-istiod-ca-cert\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:25.951407 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:25.951393 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-credential-socket\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:25.951524 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:25.951509 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvsps\" (UniqueName: \"kubernetes.io/projected/21eb266b-7a0d-43d6-b601-365ad87bec7b-kube-api-access-xvsps\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:25.951693 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:25.951680 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-workload-socket\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:25.951825 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:25.951813 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-envoy\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:25.951941 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:25.951928 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-token\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:25.952048 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:25.952037 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-workload-certs\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.052886 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.052853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-workload-socket\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.053052 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.052893 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-envoy\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.053052 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.052912 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-token\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.053052 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.053037 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-workload-certs\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.053246 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.053087 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-podinfo\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.053246 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.053150 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-data\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.053246 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.053177 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21eb266b-7a0d-43d6-b601-365ad87bec7b-istiod-ca-cert\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.053246 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.053214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-credential-socket\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.053246 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.053245 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvsps\" (UniqueName: \"kubernetes.io/projected/21eb266b-7a0d-43d6-b601-365ad87bec7b-kube-api-access-xvsps\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.053504 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.053381 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-workload-socket\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.053504 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.053426 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-workload-certs\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.053726 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.053708 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-credential-socket\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.053792 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.053753 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-data\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.054224 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.054196 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21eb266b-7a0d-43d6-b601-365ad87bec7b-istiod-ca-cert\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.055279 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.055262 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-envoy\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.055707 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.055682 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-podinfo\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.067304 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.067275 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvsps\" (UniqueName: \"kubernetes.io/projected/21eb266b-7a0d-43d6-b601-365ad87bec7b-kube-api-access-xvsps\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.067714 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.067694 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-token\") pod \"e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.149277 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.149245 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:26.303214 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.303185 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc"] Jul 16 17:22:26.305686 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:22:26.305658 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21eb266b_7a0d_43d6_b601_365ad87bec7b.slice/crio-3a371d775d1c8b37254484aa32d0f741006cb1b0f5d66ad58343d5428fc0479c WatchSource:0}: Error finding container 3a371d775d1c8b37254484aa32d0f741006cb1b0f5d66ad58343d5428fc0479c: Status 404 returned error can't find the container with id 3a371d775d1c8b37254484aa32d0f741006cb1b0f5d66ad58343d5428fc0479c Jul 16 17:22:26.308392 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.308363 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:22:26.308472 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.308427 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:22:26.308472 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.308452 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:22:26.367240 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.367216 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:26.872399 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.872363 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" event={"ID":"21eb266b-7a0d-43d6-b601-365ad87bec7b","Type":"ContainerStarted","Data":"82268001ad83cdeeb375f22b974e49dca8472b03c78736f416b2cc95b4af892c"} Jul 16 17:22:26.872576 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:26.872415 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" event={"ID":"21eb266b-7a0d-43d6-b601-365ad87bec7b","Type":"ContainerStarted","Data":"3a371d775d1c8b37254484aa32d0f741006cb1b0f5d66ad58343d5428fc0479c"} Jul 16 17:22:27.149917 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:27.149833 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:27.154330 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:27.154309 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:27.184131 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:27.184067 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" podStartSLOduration=2.184052809 podStartE2EDuration="2.184052809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:22:26.90905406 +0000 UTC m=+1884.427779294" watchObservedRunningTime="2026-07-16 17:22:27.184052809 +0000 UTC m=+1884.702778043" Jul 16 17:22:27.876986 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:27.876953 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:27.877931 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:27.877904 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:27.891076 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:27.891050 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:27.993000 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:27.992966 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:28.171891 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:28.171821 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57"] Jul 16 17:22:28.175238 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:28.175221 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" Jul 16 17:22:28.182014 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:28.181964 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-901aa3-serving-cert\"" Jul 16 17:22:28.191826 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:28.191796 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57"] Jul 16 17:22:28.225423 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:28.225386 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:28.274282 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:28.274248 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mkld\" (UniqueName: \"kubernetes.io/projected/11f75cab-2317-4b9e-853d-d54f4109fc10-kube-api-access-9mkld\") pod \"maas-api-e2e-inf-b-901aa3-677f78f768-9mv57\" (UID: \"11f75cab-2317-4b9e-853d-d54f4109fc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" Jul 16 17:22:28.274460 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:28.274316 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f75cab-2317-4b9e-853d-d54f4109fc10-maas-api-tls\") pod \"maas-api-e2e-inf-b-901aa3-677f78f768-9mv57\" (UID: \"11f75cab-2317-4b9e-853d-d54f4109fc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" Jul 16 17:22:28.274460 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:28.274343 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f75cab-2317-4b9e-853d-d54f4109fc10-openshift-service-ca\") pod \"maas-api-e2e-inf-b-901aa3-677f78f768-9mv57\" (UID: \"11f75cab-2317-4b9e-853d-d54f4109fc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" Jul 16 17:22:28.374863 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:28.374833 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mkld\" (UniqueName: \"kubernetes.io/projected/11f75cab-2317-4b9e-853d-d54f4109fc10-kube-api-access-9mkld\") pod \"maas-api-e2e-inf-b-901aa3-677f78f768-9mv57\" (UID: \"11f75cab-2317-4b9e-853d-d54f4109fc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" Jul 16 17:22:28.375040 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:28.374880 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f75cab-2317-4b9e-853d-d54f4109fc10-maas-api-tls\") pod \"maas-api-e2e-inf-b-901aa3-677f78f768-9mv57\" (UID: \"11f75cab-2317-4b9e-853d-d54f4109fc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" Jul 16 17:22:28.375040 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:28.374918 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f75cab-2317-4b9e-853d-d54f4109fc10-openshift-service-ca\") pod \"maas-api-e2e-inf-b-901aa3-677f78f768-9mv57\" (UID: \"11f75cab-2317-4b9e-853d-d54f4109fc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" Jul 16 17:22:28.375220 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:22:28.375200 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-serving-cert: secret "maas-api-e2e-inf-b-901aa3-serving-cert" not found Jul 16 17:22:28.375304 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:22:28.375282 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11f75cab-2317-4b9e-853d-d54f4109fc10-maas-api-tls podName:11f75cab-2317-4b9e-853d-d54f4109fc10 nodeName:}" failed. No retries permitted until 2026-07-16 17:22:28.875258652 +0000 UTC m=+1886.393983869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/11f75cab-2317-4b9e-853d-d54f4109fc10-maas-api-tls") pod "maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" (UID: "11f75cab-2317-4b9e-853d-d54f4109fc10") : secret "maas-api-e2e-inf-b-901aa3-serving-cert" not found Jul 16 17:22:28.375571 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:28.375551 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f75cab-2317-4b9e-853d-d54f4109fc10-openshift-service-ca\") pod \"maas-api-e2e-inf-b-901aa3-677f78f768-9mv57\" (UID: \"11f75cab-2317-4b9e-853d-d54f4109fc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" Jul 16 17:22:28.385775 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:28.385757 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mkld\" (UniqueName: \"kubernetes.io/projected/11f75cab-2317-4b9e-853d-d54f4109fc10-kube-api-access-9mkld\") pod \"maas-api-e2e-inf-b-901aa3-677f78f768-9mv57\" (UID: \"11f75cab-2317-4b9e-853d-d54f4109fc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" Jul 16 17:22:28.478226 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:28.478148 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:22:28.606201 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:28.606171 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:28.878386 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:28.878358 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f75cab-2317-4b9e-853d-d54f4109fc10-maas-api-tls\") pod \"maas-api-e2e-inf-b-901aa3-677f78f768-9mv57\" (UID: \"11f75cab-2317-4b9e-853d-d54f4109fc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" Jul 16 17:22:28.880862 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:28.880828 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f75cab-2317-4b9e-853d-d54f4109fc10-maas-api-tls\") pod \"maas-api-e2e-inf-b-901aa3-677f78f768-9mv57\" (UID: \"11f75cab-2317-4b9e-853d-d54f4109fc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" Jul 16 17:22:29.085816 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:29.085788 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" Jul 16 17:22:29.235164 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:29.235137 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57"] Jul 16 17:22:29.237721 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:22:29.237696 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f75cab_2317_4b9e_853d_d54f4109fc10.slice/crio-849017d4bccdb19b89e239c36b4029fb3bf8919feca5cdc59451e0f8d286512e WatchSource:0}: Error finding container 849017d4bccdb19b89e239c36b4029fb3bf8919feca5cdc59451e0f8d286512e: Status 404 returned error can't find the container with id 849017d4bccdb19b89e239c36b4029fb3bf8919feca5cdc59451e0f8d286512e Jul 16 17:22:29.865633 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:29.865600 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" Jul 16 17:22:29.886403 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:29.886369 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" event={"ID":"11f75cab-2317-4b9e-853d-d54f4109fc10","Type":"ContainerStarted","Data":"e8a176c270f8897d7204971fb540a31b70ab94577ec6a5ba80d05e066b8240fd"} Jul 16 17:22:29.886565 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:29.886427 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" event={"ID":"11f75cab-2317-4b9e-853d-d54f4109fc10","Type":"ContainerStarted","Data":"849017d4bccdb19b89e239c36b4029fb3bf8919feca5cdc59451e0f8d286512e"} Jul 16 17:22:29.886565 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:29.886458 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" Jul 16 17:22:30.007238 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:30.007161 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" podStartSLOduration=1.6454809689999998 podStartE2EDuration="2.007140295s" totalImagesPullingTime="361.659326ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:22:30.006210601 +0000 UTC m=+1887.524935836" watchObservedRunningTime="2026-07-16 17:22:30.007140295 +0000 UTC m=+1887.525865529" Jul 16 17:22:30.658746 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:30.658707 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:31.076507 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:31.076386 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:35.118947 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:35.118910 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:35.896848 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:35.896819 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" Jul 16 17:22:42.237426 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:42.237386 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:22:43.223184 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:43.223153 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:43.349775 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:43.349745 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:43.919317 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:43.919274 2575 patch_prober.go:28] interesting pod/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 16 17:22:43.919497 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:43.919339 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" podUID="21eb266b-7a0d-43d6-b601-365ad87bec7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 17:22:44.961233 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:44.961203 2575 generic.go:356] "Generic (PLEG): container finished" podID="21eb266b-7a0d-43d6-b601-365ad87bec7b" containerID="82268001ad83cdeeb375f22b974e49dca8472b03c78736f416b2cc95b4af892c" exitCode=0 Jul 16 17:22:44.961637 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:44.961280 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" event={"ID":"21eb266b-7a0d-43d6-b601-365ad87bec7b","Type":"ContainerDied","Data":"82268001ad83cdeeb375f22b974e49dca8472b03c78736f416b2cc95b4af892c"} Jul 16 17:22:44.961684 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:44.961661 2575 scope.go:117] "RemoveContainer" containerID="82268001ad83cdeeb375f22b974e49dca8472b03c78736f416b2cc95b4af892c" Jul 16 17:22:44.962633 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:44.962608 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:22:44.962698 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:44.962674 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:22:44.962734 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:44.962708 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:22:45.967304 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:45.967269 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" event={"ID":"21eb266b-7a0d-43d6-b601-365ad87bec7b","Type":"ContainerStarted","Data":"ea1b9cd7ce77a66a5f13b3045160e0e37e6d0e382d813f8294beef4a9bb2267c"} Jul 16 17:22:46.149514 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:46.149475 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:46.149824 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:46.149802 2575 patch_prober.go:28] interesting pod/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 16 17:22:46.149877 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:46.149858 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" podUID="21eb266b-7a0d-43d6-b601-365ad87bec7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 16 17:22:47.149854 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:47.149820 2575 patch_prober.go:28] interesting pod/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 16 17:22:47.150245 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:47.149871 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" podUID="21eb266b-7a0d-43d6-b601-365ad87bec7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 16 17:22:48.150448 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:48.150415 2575 patch_prober.go:28] interesting pod/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 16 17:22:48.150919 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:48.150469 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" podUID="21eb266b-7a0d-43d6-b601-365ad87bec7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 16 17:22:49.150523 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:49.150487 2575 patch_prober.go:28] interesting pod/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 16 17:22:49.150913 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:49.150553 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" podUID="21eb266b-7a0d-43d6-b601-365ad87bec7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 16 17:22:49.975335 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:49.975304 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:50.155086 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:50.155048 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:50.155810 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:50.155784 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:50.158942 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:50.158887 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:22:51.845797 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:51.845767 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:51.958354 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:51.958325 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:22:52.772703 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:22:52.772652 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:23:03.920011 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:23:03.919977 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:23:26.208311 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:23:26.208267 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:23:30.491275 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:23:30.491243 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:23:55.384233 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:23:55.384153 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:24:16.713649 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:24:16.713616 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:24:17.966005 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:24:17.965969 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:24:18.438766 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:24:18.438732 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:24:21.379712 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:24:21.379681 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:24:23.507296 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:24:23.507256 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:01.844016 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:01.843964 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp"] Jul 16 17:25:01.844634 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:01.844307 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" podUID="cbc3aa04-255b-4957-aa36-d5784fe138df" containerName="maas-api" containerID="cri-o://45e9862b2072b8928271497bfbfc115520235bf97ee0f93bf06755302b5c473a" gracePeriod=30 Jul 16 17:25:01.894038 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:01.894004 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:02.098075 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.097998 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" Jul 16 17:25:02.257873 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.257833 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc3aa04-255b-4957-aa36-d5784fe138df-openshift-service-ca\") pod \"cbc3aa04-255b-4957-aa36-d5784fe138df\" (UID: \"cbc3aa04-255b-4957-aa36-d5784fe138df\") " Jul 16 17:25:02.258049 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.257888 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc3aa04-255b-4957-aa36-d5784fe138df-maas-api-tls\") pod \"cbc3aa04-255b-4957-aa36-d5784fe138df\" (UID: \"cbc3aa04-255b-4957-aa36-d5784fe138df\") " Jul 16 17:25:02.258049 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.257918 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snvmq\" (UniqueName: \"kubernetes.io/projected/cbc3aa04-255b-4957-aa36-d5784fe138df-kube-api-access-snvmq\") pod \"cbc3aa04-255b-4957-aa36-d5784fe138df\" (UID: \"cbc3aa04-255b-4957-aa36-d5784fe138df\") " Jul 16 17:25:02.258303 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.258281 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc3aa04-255b-4957-aa36-d5784fe138df-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cbc3aa04-255b-4957-aa36-d5784fe138df" (UID: "cbc3aa04-255b-4957-aa36-d5784fe138df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:25:02.260138 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.260107 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc3aa04-255b-4957-aa36-d5784fe138df-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cbc3aa04-255b-4957-aa36-d5784fe138df" (UID: "cbc3aa04-255b-4957-aa36-d5784fe138df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:25:02.260261 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.260203 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc3aa04-255b-4957-aa36-d5784fe138df-kube-api-access-snvmq" (OuterVolumeSpecName: "kube-api-access-snvmq") pod "cbc3aa04-255b-4957-aa36-d5784fe138df" (UID: "cbc3aa04-255b-4957-aa36-d5784fe138df"). InnerVolumeSpecName "kube-api-access-snvmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:25:02.358637 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.358601 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc3aa04-255b-4957-aa36-d5784fe138df-openshift-service-ca\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:25:02.358637 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.358632 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc3aa04-255b-4957-aa36-d5784fe138df-maas-api-tls\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:25:02.358637 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.358645 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snvmq\" (UniqueName: \"kubernetes.io/projected/cbc3aa04-255b-4957-aa36-d5784fe138df-kube-api-access-snvmq\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:25:02.604107 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.604071 2575 generic.go:356] "Generic (PLEG): container finished" podID="cbc3aa04-255b-4957-aa36-d5784fe138df" containerID="45e9862b2072b8928271497bfbfc115520235bf97ee0f93bf06755302b5c473a" exitCode=0 Jul 16 17:25:02.604282 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.604142 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" Jul 16 17:25:02.604282 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.604137 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" event={"ID":"cbc3aa04-255b-4957-aa36-d5784fe138df","Type":"ContainerDied","Data":"45e9862b2072b8928271497bfbfc115520235bf97ee0f93bf06755302b5c473a"} Jul 16 17:25:02.604282 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.604202 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp" event={"ID":"cbc3aa04-255b-4957-aa36-d5784fe138df","Type":"ContainerDied","Data":"a6e0a90ce1b613245cbf786686eda954459922070a5c9206bb7863eef296f231"} Jul 16 17:25:02.604282 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.604234 2575 scope.go:117] "RemoveContainer" containerID="45e9862b2072b8928271497bfbfc115520235bf97ee0f93bf06755302b5c473a" Jul 16 17:25:02.617228 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.617206 2575 scope.go:117] "RemoveContainer" containerID="45e9862b2072b8928271497bfbfc115520235bf97ee0f93bf06755302b5c473a" Jul 16 17:25:02.617518 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:25:02.617498 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45e9862b2072b8928271497bfbfc115520235bf97ee0f93bf06755302b5c473a\": container with ID starting with 45e9862b2072b8928271497bfbfc115520235bf97ee0f93bf06755302b5c473a not found: ID does not exist" containerID="45e9862b2072b8928271497bfbfc115520235bf97ee0f93bf06755302b5c473a" Jul 16 17:25:02.617577 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.617529 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45e9862b2072b8928271497bfbfc115520235bf97ee0f93bf06755302b5c473a"} err="failed to get container status \"45e9862b2072b8928271497bfbfc115520235bf97ee0f93bf06755302b5c473a\": rpc error: code = NotFound desc = could not find container \"45e9862b2072b8928271497bfbfc115520235bf97ee0f93bf06755302b5c473a\": container with ID starting with 45e9862b2072b8928271497bfbfc115520235bf97ee0f93bf06755302b5c473a not found: ID does not exist" Jul 16 17:25:02.648084 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.648050 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp"] Jul 16 17:25:02.652228 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.652204 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7b9f8f-599fc74f88-6x6wp"] Jul 16 17:25:02.909842 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:02.909744 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbc3aa04-255b-4957-aa36-d5784fe138df" path="/var/lib/kubelet/pods/cbc3aa04-255b-4957-aa36-d5784fe138df/volumes" Jul 16 17:25:03.094186 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:03.094147 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:05.408952 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:05.408915 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:25:11.261655 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:11.261616 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:11.412752 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:11.412720 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:11.705637 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:11.705604 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:12.318979 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:12.318940 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:12.931651 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:12.931624 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:19.916546 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:19.916516 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:20.037494 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.037464 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:20.140374 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.140339 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:20.352141 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.352106 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57"] Jul 16 17:25:20.352413 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.352359 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" podUID="11f75cab-2317-4b9e-853d-d54f4109fc10" containerName="maas-api" containerID="cri-o://e8a176c270f8897d7204971fb540a31b70ab94577ec6a5ba80d05e066b8240fd" gracePeriod=30 Jul 16 17:25:20.419213 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.419181 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:20.608795 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.608737 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" Jul 16 17:25:20.693751 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.693718 2575 generic.go:356] "Generic (PLEG): container finished" podID="11f75cab-2317-4b9e-853d-d54f4109fc10" containerID="e8a176c270f8897d7204971fb540a31b70ab94577ec6a5ba80d05e066b8240fd" exitCode=0 Jul 16 17:25:20.693918 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.693774 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" Jul 16 17:25:20.693918 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.693812 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" event={"ID":"11f75cab-2317-4b9e-853d-d54f4109fc10","Type":"ContainerDied","Data":"e8a176c270f8897d7204971fb540a31b70ab94577ec6a5ba80d05e066b8240fd"} Jul 16 17:25:20.693918 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.693891 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57" event={"ID":"11f75cab-2317-4b9e-853d-d54f4109fc10","Type":"ContainerDied","Data":"849017d4bccdb19b89e239c36b4029fb3bf8919feca5cdc59451e0f8d286512e"} Jul 16 17:25:20.693918 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.693917 2575 scope.go:117] "RemoveContainer" containerID="e8a176c270f8897d7204971fb540a31b70ab94577ec6a5ba80d05e066b8240fd" Jul 16 17:25:20.701985 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.701965 2575 scope.go:117] "RemoveContainer" containerID="e8a176c270f8897d7204971fb540a31b70ab94577ec6a5ba80d05e066b8240fd" Jul 16 17:25:20.702266 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:25:20.702245 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a176c270f8897d7204971fb540a31b70ab94577ec6a5ba80d05e066b8240fd\": container with ID starting with e8a176c270f8897d7204971fb540a31b70ab94577ec6a5ba80d05e066b8240fd not found: ID does not exist" containerID="e8a176c270f8897d7204971fb540a31b70ab94577ec6a5ba80d05e066b8240fd" Jul 16 17:25:20.702344 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.702277 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a176c270f8897d7204971fb540a31b70ab94577ec6a5ba80d05e066b8240fd"} err="failed to get container status \"e8a176c270f8897d7204971fb540a31b70ab94577ec6a5ba80d05e066b8240fd\": rpc error: code = NotFound desc = could not find container \"e8a176c270f8897d7204971fb540a31b70ab94577ec6a5ba80d05e066b8240fd\": container with ID starting with e8a176c270f8897d7204971fb540a31b70ab94577ec6a5ba80d05e066b8240fd not found: ID does not exist" Jul 16 17:25:20.711492 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.711473 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f75cab-2317-4b9e-853d-d54f4109fc10-openshift-service-ca\") pod \"11f75cab-2317-4b9e-853d-d54f4109fc10\" (UID: \"11f75cab-2317-4b9e-853d-d54f4109fc10\") " Jul 16 17:25:20.711589 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.711511 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mkld\" (UniqueName: \"kubernetes.io/projected/11f75cab-2317-4b9e-853d-d54f4109fc10-kube-api-access-9mkld\") pod \"11f75cab-2317-4b9e-853d-d54f4109fc10\" (UID: \"11f75cab-2317-4b9e-853d-d54f4109fc10\") " Jul 16 17:25:20.711589 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.711536 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f75cab-2317-4b9e-853d-d54f4109fc10-maas-api-tls\") pod \"11f75cab-2317-4b9e-853d-d54f4109fc10\" (UID: \"11f75cab-2317-4b9e-853d-d54f4109fc10\") " Jul 16 17:25:20.711876 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.711853 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f75cab-2317-4b9e-853d-d54f4109fc10-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "11f75cab-2317-4b9e-853d-d54f4109fc10" (UID: "11f75cab-2317-4b9e-853d-d54f4109fc10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:25:20.713495 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.713471 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f75cab-2317-4b9e-853d-d54f4109fc10-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "11f75cab-2317-4b9e-853d-d54f4109fc10" (UID: "11f75cab-2317-4b9e-853d-d54f4109fc10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:25:20.713644 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.713624 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f75cab-2317-4b9e-853d-d54f4109fc10-kube-api-access-9mkld" (OuterVolumeSpecName: "kube-api-access-9mkld") pod "11f75cab-2317-4b9e-853d-d54f4109fc10" (UID: "11f75cab-2317-4b9e-853d-d54f4109fc10"). InnerVolumeSpecName "kube-api-access-9mkld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:25:20.813153 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.813091 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f75cab-2317-4b9e-853d-d54f4109fc10-openshift-service-ca\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:25:20.813153 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.813148 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mkld\" (UniqueName: \"kubernetes.io/projected/11f75cab-2317-4b9e-853d-d54f4109fc10-kube-api-access-9mkld\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:25:20.813153 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:20.813158 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f75cab-2317-4b9e-853d-d54f4109fc10-maas-api-tls\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:25:21.029052 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:21.029011 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57"] Jul 16 17:25:21.030912 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:21.030888 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-901aa3-677f78f768-9mv57"] Jul 16 17:25:21.927209 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:21.927176 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:22.909900 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:22.909860 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11f75cab-2317-4b9e-853d-d54f4109fc10" path="/var/lib/kubelet/pods/11f75cab-2317-4b9e-853d-d54f4109fc10/volumes" Jul 16 17:25:23.315894 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:23.315820 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:25:30.417544 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:30.417466 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:30.613901 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:30.613868 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:30.817342 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:30.817262 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc"] Jul 16 17:25:30.817495 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:30.817470 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" podUID="21eb266b-7a0d-43d6-b601-365ad87bec7b" containerName="istio-proxy" containerID="cri-o://ea1b9cd7ce77a66a5f13b3045160e0e37e6d0e382d813f8294beef4a9bb2267c" gracePeriod=30 Jul 16 17:25:30.831693 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:30.831661 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:31.141074 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:31.141047 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:31.626330 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:31.626293 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:32.261161 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:32.261128 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:32.417515 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:32.417486 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:34.729336 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:34.729298 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:34.874275 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:34.874238 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q"] Jul 16 17:25:34.874809 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:34.874793 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f75cab-2317-4b9e-853d-d54f4109fc10" containerName="maas-api" Jul 16 17:25:34.874870 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:34.874812 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f75cab-2317-4b9e-853d-d54f4109fc10" containerName="maas-api" Jul 16 17:25:34.874870 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:34.874825 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc3aa04-255b-4957-aa36-d5784fe138df" containerName="maas-api" Jul 16 17:25:34.874870 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:34.874834 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc3aa04-255b-4957-aa36-d5784fe138df" containerName="maas-api" Jul 16 17:25:34.874964 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:34.874918 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="11f75cab-2317-4b9e-853d-d54f4109fc10" containerName="maas-api" Jul 16 17:25:34.874964 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:34.874931 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc3aa04-255b-4957-aa36-d5784fe138df" containerName="maas-api" Jul 16 17:25:34.879369 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:34.879349 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" Jul 16 17:25:34.885022 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:34.885002 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bff62\"" Jul 16 17:25:34.885771 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:34.885608 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 17:25:34.886143 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:34.886125 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 17:25:34.896908 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:34.896889 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-36dc65-serving-cert\"" Jul 16 17:25:34.923022 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:34.922998 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q"] Jul 16 17:25:34.928412 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:34.928389 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:34.940717 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:34.940685 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a47e04-2b57-441a-90ac-2887631e94e9-maas-api-tls\") pod \"maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q\" (UID: \"b0a47e04-2b57-441a-90ac-2887631e94e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" Jul 16 17:25:34.940876 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:34.940837 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a47e04-2b57-441a-90ac-2887631e94e9-openshift-service-ca\") pod \"maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q\" (UID: \"b0a47e04-2b57-441a-90ac-2887631e94e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" Jul 16 17:25:34.940945 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:34.940904 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96r7z\" (UniqueName: \"kubernetes.io/projected/b0a47e04-2b57-441a-90ac-2887631e94e9-kube-api-access-96r7z\") pod \"maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q\" (UID: \"b0a47e04-2b57-441a-90ac-2887631e94e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" Jul 16 17:25:35.041953 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:35.041874 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96r7z\" (UniqueName: \"kubernetes.io/projected/b0a47e04-2b57-441a-90ac-2887631e94e9-kube-api-access-96r7z\") pod \"maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q\" (UID: \"b0a47e04-2b57-441a-90ac-2887631e94e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" Jul 16 17:25:35.041953 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:35.041919 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a47e04-2b57-441a-90ac-2887631e94e9-maas-api-tls\") pod \"maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q\" (UID: \"b0a47e04-2b57-441a-90ac-2887631e94e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" Jul 16 17:25:35.042161 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:35.041974 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a47e04-2b57-441a-90ac-2887631e94e9-openshift-service-ca\") pod \"maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q\" (UID: \"b0a47e04-2b57-441a-90ac-2887631e94e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" Jul 16 17:25:35.042576 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:35.042550 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a47e04-2b57-441a-90ac-2887631e94e9-openshift-service-ca\") pod \"maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q\" (UID: \"b0a47e04-2b57-441a-90ac-2887631e94e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" Jul 16 17:25:35.044544 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:35.044517 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a47e04-2b57-441a-90ac-2887631e94e9-maas-api-tls\") pod \"maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q\" (UID: \"b0a47e04-2b57-441a-90ac-2887631e94e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" Jul 16 17:25:35.069183 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:35.069163 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96r7z\" (UniqueName: \"kubernetes.io/projected/b0a47e04-2b57-441a-90ac-2887631e94e9-kube-api-access-96r7z\") pod \"maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q\" (UID: \"b0a47e04-2b57-441a-90ac-2887631e94e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" Jul 16 17:25:35.103474 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:35.103450 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:35.158525 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:35.158491 2575 patch_prober.go:28] interesting pod/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 17:25:35.158643 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:35.158556 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" podUID="21eb266b-7a0d-43d6-b601-365ad87bec7b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:25:35.189003 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:35.188978 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" Jul 16 17:25:35.243058 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:35.243032 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:35.341631 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:35.341607 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q"] Jul 16 17:25:35.344342 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:25:35.344308 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a47e04_2b57_441a_90ac_2887631e94e9.slice/crio-36955bc0ac0f4b3b56e2aa59812c22be9dbfd40fb435b9f5c764a9a127792938 WatchSource:0}: Error finding container 36955bc0ac0f4b3b56e2aa59812c22be9dbfd40fb435b9f5c764a9a127792938: Status 404 returned error can't find the container with id 36955bc0ac0f4b3b56e2aa59812c22be9dbfd40fb435b9f5c764a9a127792938 Jul 16 17:25:35.345769 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:35.345747 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:25:35.760158 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:35.760123 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" event={"ID":"b0a47e04-2b57-441a-90ac-2887631e94e9","Type":"ContainerStarted","Data":"36955bc0ac0f4b3b56e2aa59812c22be9dbfd40fb435b9f5c764a9a127792938"} Jul 16 17:25:36.058714 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.058689 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:36.084730 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.084707 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:25:36.152902 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.152873 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-workload-socket\") pod \"21eb266b-7a0d-43d6-b601-365ad87bec7b\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " Jul 16 17:25:36.153070 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.152945 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-workload-certs\") pod \"21eb266b-7a0d-43d6-b601-365ad87bec7b\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " Jul 16 17:25:36.153070 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.152993 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-podinfo\") pod \"21eb266b-7a0d-43d6-b601-365ad87bec7b\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " Jul 16 17:25:36.153070 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.153027 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-data\") pod \"21eb266b-7a0d-43d6-b601-365ad87bec7b\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " Jul 16 17:25:36.153070 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.153043 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-envoy\") pod \"21eb266b-7a0d-43d6-b601-365ad87bec7b\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " Jul 16 17:25:36.153304 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.153073 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21eb266b-7a0d-43d6-b601-365ad87bec7b-istiod-ca-cert\") pod \"21eb266b-7a0d-43d6-b601-365ad87bec7b\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " Jul 16 17:25:36.153304 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.153165 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-credential-socket\") pod \"21eb266b-7a0d-43d6-b601-365ad87bec7b\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " Jul 16 17:25:36.153304 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.153189 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "21eb266b-7a0d-43d6-b601-365ad87bec7b" (UID: "21eb266b-7a0d-43d6-b601-365ad87bec7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:25:36.153304 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.153202 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-token\") pod \"21eb266b-7a0d-43d6-b601-365ad87bec7b\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " Jul 16 17:25:36.153304 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.153248 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "21eb266b-7a0d-43d6-b601-365ad87bec7b" (UID: "21eb266b-7a0d-43d6-b601-365ad87bec7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:25:36.153565 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.153306 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvsps\" (UniqueName: \"kubernetes.io/projected/21eb266b-7a0d-43d6-b601-365ad87bec7b-kube-api-access-xvsps\") pod \"21eb266b-7a0d-43d6-b601-365ad87bec7b\" (UID: \"21eb266b-7a0d-43d6-b601-365ad87bec7b\") " Jul 16 17:25:36.153621 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.153567 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "21eb266b-7a0d-43d6-b601-365ad87bec7b" (UID: "21eb266b-7a0d-43d6-b601-365ad87bec7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:25:36.153621 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.153592 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21eb266b-7a0d-43d6-b601-365ad87bec7b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "21eb266b-7a0d-43d6-b601-365ad87bec7b" (UID: "21eb266b-7a0d-43d6-b601-365ad87bec7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:25:36.153621 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.153600 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-workload-socket\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:25:36.153766 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.153632 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-workload-certs\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:25:36.154024 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.154002 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-data" (OuterVolumeSpecName: "istio-data") pod "21eb266b-7a0d-43d6-b601-365ad87bec7b" (UID: "21eb266b-7a0d-43d6-b601-365ad87bec7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:25:36.155348 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.155327 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-token" (OuterVolumeSpecName: "istio-token") pod "21eb266b-7a0d-43d6-b601-365ad87bec7b" (UID: "21eb266b-7a0d-43d6-b601-365ad87bec7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:25:36.155426 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.155396 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "21eb266b-7a0d-43d6-b601-365ad87bec7b" (UID: "21eb266b-7a0d-43d6-b601-365ad87bec7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 17:25:36.155744 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.155727 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21eb266b-7a0d-43d6-b601-365ad87bec7b-kube-api-access-xvsps" (OuterVolumeSpecName: "kube-api-access-xvsps") pod "21eb266b-7a0d-43d6-b601-365ad87bec7b" (UID: "21eb266b-7a0d-43d6-b601-365ad87bec7b"). InnerVolumeSpecName "kube-api-access-xvsps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:25:36.155744 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.155731 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "21eb266b-7a0d-43d6-b601-365ad87bec7b" (UID: "21eb266b-7a0d-43d6-b601-365ad87bec7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:25:36.202060 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.202029 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:36.254661 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.254625 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvsps\" (UniqueName: \"kubernetes.io/projected/21eb266b-7a0d-43d6-b601-365ad87bec7b-kube-api-access-xvsps\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:25:36.254661 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.254659 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-podinfo\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:25:36.254661 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.254670 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-data\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:25:36.254894 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.254678 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-envoy\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:25:36.254894 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.254686 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21eb266b-7a0d-43d6-b601-365ad87bec7b-istiod-ca-cert\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:25:36.254894 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.254696 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21eb266b-7a0d-43d6-b601-365ad87bec7b-credential-socket\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:25:36.254894 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.254706 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21eb266b-7a0d-43d6-b601-365ad87bec7b-istio-token\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:25:36.766482 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.766444 2575 generic.go:356] "Generic (PLEG): container finished" podID="21eb266b-7a0d-43d6-b601-365ad87bec7b" containerID="ea1b9cd7ce77a66a5f13b3045160e0e37e6d0e382d813f8294beef4a9bb2267c" exitCode=0 Jul 16 17:25:36.766883 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.766494 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" Jul 16 17:25:36.766883 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.766532 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" event={"ID":"21eb266b-7a0d-43d6-b601-365ad87bec7b","Type":"ContainerDied","Data":"ea1b9cd7ce77a66a5f13b3045160e0e37e6d0e382d813f8294beef4a9bb2267c"} Jul 16 17:25:36.766883 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.766584 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc" event={"ID":"21eb266b-7a0d-43d6-b601-365ad87bec7b","Type":"ContainerDied","Data":"3a371d775d1c8b37254484aa32d0f741006cb1b0f5d66ad58343d5428fc0479c"} Jul 16 17:25:36.766883 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.766610 2575 scope.go:117] "RemoveContainer" containerID="ea1b9cd7ce77a66a5f13b3045160e0e37e6d0e382d813f8294beef4a9bb2267c" Jul 16 17:25:36.775836 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.775821 2575 scope.go:117] "RemoveContainer" containerID="82268001ad83cdeeb375f22b974e49dca8472b03c78736f416b2cc95b4af892c" Jul 16 17:25:36.783416 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.783400 2575 scope.go:117] "RemoveContainer" containerID="ea1b9cd7ce77a66a5f13b3045160e0e37e6d0e382d813f8294beef4a9bb2267c" Jul 16 17:25:36.783687 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:25:36.783666 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea1b9cd7ce77a66a5f13b3045160e0e37e6d0e382d813f8294beef4a9bb2267c\": container with ID starting with ea1b9cd7ce77a66a5f13b3045160e0e37e6d0e382d813f8294beef4a9bb2267c not found: ID does not exist" containerID="ea1b9cd7ce77a66a5f13b3045160e0e37e6d0e382d813f8294beef4a9bb2267c" Jul 16 17:25:36.783772 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.783699 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1b9cd7ce77a66a5f13b3045160e0e37e6d0e382d813f8294beef4a9bb2267c"} err="failed to get container status \"ea1b9cd7ce77a66a5f13b3045160e0e37e6d0e382d813f8294beef4a9bb2267c\": rpc error: code = NotFound desc = could not find container \"ea1b9cd7ce77a66a5f13b3045160e0e37e6d0e382d813f8294beef4a9bb2267c\": container with ID starting with ea1b9cd7ce77a66a5f13b3045160e0e37e6d0e382d813f8294beef4a9bb2267c not found: ID does not exist" Jul 16 17:25:36.783772 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.783724 2575 scope.go:117] "RemoveContainer" containerID="82268001ad83cdeeb375f22b974e49dca8472b03c78736f416b2cc95b4af892c" Jul 16 17:25:36.783981 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:25:36.783965 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82268001ad83cdeeb375f22b974e49dca8472b03c78736f416b2cc95b4af892c\": container with ID starting with 82268001ad83cdeeb375f22b974e49dca8472b03c78736f416b2cc95b4af892c not found: ID does not exist" containerID="82268001ad83cdeeb375f22b974e49dca8472b03c78736f416b2cc95b4af892c" Jul 16 17:25:36.784034 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.783991 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82268001ad83cdeeb375f22b974e49dca8472b03c78736f416b2cc95b4af892c"} err="failed to get container status \"82268001ad83cdeeb375f22b974e49dca8472b03c78736f416b2cc95b4af892c\": rpc error: code = NotFound desc = could not find container \"82268001ad83cdeeb375f22b974e49dca8472b03c78736f416b2cc95b4af892c\": container with ID starting with 82268001ad83cdeeb375f22b974e49dca8472b03c78736f416b2cc95b4af892c not found: ID does not exist" Jul 16 17:25:36.807618 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.807594 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc"] Jul 16 17:25:36.823877 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.823847 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-901aa3-openshift-default-f98c4d7b5-82dzc"] Jul 16 17:25:36.909561 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:36.909529 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21eb266b-7a0d-43d6-b601-365ad87bec7b" path="/var/lib/kubelet/pods/21eb266b-7a0d-43d6-b601-365ad87bec7b/volumes" Jul 16 17:25:37.774084 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:37.774050 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" event={"ID":"b0a47e04-2b57-441a-90ac-2887631e94e9","Type":"ContainerStarted","Data":"d30c78f7088d6fdbad0788f8f0aa1ef2ab7df6c06046272d5ee34400a52fc1c4"} Jul 16 17:25:37.774487 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:37.774154 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" Jul 16 17:25:37.817303 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:37.817254 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" podStartSLOduration=2.14177652 podStartE2EDuration="3.817240827s" totalImagesPullingTime="1.675464307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:25:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:25:37.807541194 +0000 UTC m=+2075.326266427" watchObservedRunningTime="2026-07-16 17:25:37.817240827 +0000 UTC m=+2075.335966060" Jul 16 17:25:40.342975 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:40.342943 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:40.901001 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:40.900971 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:25:40.998919 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:40.998890 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:41.514127 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.514079 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s"] Jul 16 17:25:41.514565 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.514429 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21eb266b-7a0d-43d6-b601-365ad87bec7b" containerName="istio-proxy" Jul 16 17:25:41.514565 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.514440 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="21eb266b-7a0d-43d6-b601-365ad87bec7b" containerName="istio-proxy" Jul 16 17:25:41.514565 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.514452 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21eb266b-7a0d-43d6-b601-365ad87bec7b" containerName="istio-proxy" Jul 16 17:25:41.514565 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.514461 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="21eb266b-7a0d-43d6-b601-365ad87bec7b" containerName="istio-proxy" Jul 16 17:25:41.514750 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.514632 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="21eb266b-7a0d-43d6-b601-365ad87bec7b" containerName="istio-proxy" Jul 16 17:25:41.514750 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.514644 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="21eb266b-7a0d-43d6-b601-365ad87bec7b" containerName="istio-proxy" Jul 16 17:25:41.517967 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.517948 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.520036 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.520016 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:41.527853 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.527834 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7ca5c1-openshift-default-dockercfg-bf7hx\"" Jul 16 17:25:41.588210 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.588171 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s"] Jul 16 17:25:41.603306 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.603277 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-credential-socket\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.603465 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.603327 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-data\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.603465 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.603411 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6qwc\" (UniqueName: \"kubernetes.io/projected/d3e80a3e-3027-455e-b10d-c2fb44987f72-kube-api-access-s6qwc\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.603465 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.603453 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-podinfo\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.603592 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.603525 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-workload-certs\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.603592 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.603560 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-envoy\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.603695 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.603655 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-workload-socket\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.603745 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.603711 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e80a3e-3027-455e-b10d-c2fb44987f72-istiod-ca-cert\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.603787 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.603752 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-token\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.704665 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.704628 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6qwc\" (UniqueName: \"kubernetes.io/projected/d3e80a3e-3027-455e-b10d-c2fb44987f72-kube-api-access-s6qwc\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.704855 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.704672 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-podinfo\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.704855 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.704706 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-workload-certs\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.704855 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.704730 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-envoy\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.704855 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.704835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-workload-socket\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.705198 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.704871 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e80a3e-3027-455e-b10d-c2fb44987f72-istiod-ca-cert\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.705198 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.704898 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-token\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.705198 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.704933 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-credential-socket\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.705198 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.704974 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-data\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.705420 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.705200 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-workload-certs\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.705420 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.705225 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-workload-socket\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.705420 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.705337 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-data\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.705546 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.705496 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-credential-socket\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.705791 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.705770 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e80a3e-3027-455e-b10d-c2fb44987f72-istiod-ca-cert\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.707247 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.707217 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-podinfo\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.707339 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.707220 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-envoy\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.750521 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.750364 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6qwc\" (UniqueName: \"kubernetes.io/projected/d3e80a3e-3027-455e-b10d-c2fb44987f72-kube-api-access-s6qwc\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.814872 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.814799 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-token\") pod \"e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.828752 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.828728 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:41.922408 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.922383 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:41.982271 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.982158 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s"] Jul 16 17:25:41.984461 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:25:41.984435 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e80a3e_3027_455e_b10d_c2fb44987f72.slice/crio-d57b73385070f735c801f24ce11c3249f7877bd261d8f3b7d3bad8d6e24f971f WatchSource:0}: Error finding container d57b73385070f735c801f24ce11c3249f7877bd261d8f3b7d3bad8d6e24f971f: Status 404 returned error can't find the container with id d57b73385070f735c801f24ce11c3249f7877bd261d8f3b7d3bad8d6e24f971f Jul 16 17:25:41.986665 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.986635 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:25:41.986751 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.986697 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:25:41.986751 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:41.986727 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:25:42.797076 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:42.797042 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" event={"ID":"d3e80a3e-3027-455e-b10d-c2fb44987f72","Type":"ContainerStarted","Data":"1536ebbdb868face0c9b0cf4008ba1d729d6d89578447d6798ad8fe21700bc31"} Jul 16 17:25:42.797482 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:42.797110 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" event={"ID":"d3e80a3e-3027-455e-b10d-c2fb44987f72","Type":"ContainerStarted","Data":"d57b73385070f735c801f24ce11c3249f7877bd261d8f3b7d3bad8d6e24f971f"} Jul 16 17:25:42.829138 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:42.829087 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:42.925979 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:42.925922 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" podStartSLOduration=1.9259050549999999 podStartE2EDuration="1.925905055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:25:42.912716735 +0000 UTC m=+2080.431441970" watchObservedRunningTime="2026-07-16 17:25:42.925905055 +0000 UTC m=+2080.444630372" Jul 16 17:25:43.784200 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:43.784169 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" Jul 16 17:25:43.835153 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:43.835123 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:44.231808 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.231773 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:44.391828 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.391795 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:44.418290 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.418256 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h"] Jul 16 17:25:44.421700 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.421680 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" Jul 16 17:25:44.426921 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.426893 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-7ca5c1-serving-cert\"" Jul 16 17:25:44.453337 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.453312 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h"] Jul 16 17:25:44.529208 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.529117 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh2b8\" (UniqueName: \"kubernetes.io/projected/20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9-kube-api-access-bh2b8\") pod \"maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h\" (UID: \"20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" Jul 16 17:25:44.529208 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.529163 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9-openshift-service-ca\") pod \"maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h\" (UID: \"20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" Jul 16 17:25:44.529208 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.529182 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9-maas-api-tls\") pod \"maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h\" (UID: \"20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" Jul 16 17:25:44.568802 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.568769 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:44.629642 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.629605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh2b8\" (UniqueName: \"kubernetes.io/projected/20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9-kube-api-access-bh2b8\") pod \"maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h\" (UID: \"20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" Jul 16 17:25:44.629808 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.629648 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9-openshift-service-ca\") pod \"maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h\" (UID: \"20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" Jul 16 17:25:44.629808 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.629774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9-maas-api-tls\") pod \"maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h\" (UID: \"20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" Jul 16 17:25:44.630316 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.630297 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9-openshift-service-ca\") pod \"maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h\" (UID: \"20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" Jul 16 17:25:44.632281 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.632258 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9-maas-api-tls\") pod \"maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h\" (UID: \"20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" Jul 16 17:25:44.642727 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.642701 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh2b8\" (UniqueName: \"kubernetes.io/projected/20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9-kube-api-access-bh2b8\") pod \"maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h\" (UID: \"20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" Jul 16 17:25:44.699072 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.699045 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:44.732164 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.732144 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" Jul 16 17:25:44.806643 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.806617 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:44.807725 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.807706 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:44.909595 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:25:44.909571 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f73e9d_9fb6_4f28_a13d_8b9f2678c7d9.slice/crio-c1bd8247b86e4061205c61a3df217f0c8e1fb3be9fab44341ba85a0dab92b062 WatchSource:0}: Error finding container c1bd8247b86e4061205c61a3df217f0c8e1fb3be9fab44341ba85a0dab92b062: Status 404 returned error can't find the container with id c1bd8247b86e4061205c61a3df217f0c8e1fb3be9fab44341ba85a0dab92b062 Jul 16 17:25:44.910439 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:44.910418 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h"] Jul 16 17:25:45.478555 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:45.478525 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:45.625661 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:45.625630 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:45.812217 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:45.812131 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" event={"ID":"20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9","Type":"ContainerStarted","Data":"c1bd8247b86e4061205c61a3df217f0c8e1fb3be9fab44341ba85a0dab92b062"} Jul 16 17:25:46.818189 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:46.818153 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" event={"ID":"20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9","Type":"ContainerStarted","Data":"39ec852b4f675c40432f396bf417950e8485b7e6fd8c1a6d1654478a4fe8b282"} Jul 16 17:25:46.869349 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:46.869298 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" podStartSLOduration=1.7481937360000002 podStartE2EDuration="2.869280593s" totalImagesPullingTime="1.121086857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:25:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:25:46.868617444 +0000 UTC m=+2084.387342677" watchObservedRunningTime="2026-07-16 17:25:46.869280593 +0000 UTC m=+2084.388005806" Jul 16 17:25:47.824968 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:47.824932 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" Jul 16 17:25:49.983442 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:49.983406 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:53.837502 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:53.837425 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" Jul 16 17:25:56.666399 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:56.666365 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:56.797851 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:56.797821 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:56.875212 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:56.875183 2575 generic.go:356] "Generic (PLEG): container finished" podID="d3e80a3e-3027-455e-b10d-c2fb44987f72" containerID="1536ebbdb868face0c9b0cf4008ba1d729d6d89578447d6798ad8fe21700bc31" exitCode=0 Jul 16 17:25:56.875416 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:56.875262 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" event={"ID":"d3e80a3e-3027-455e-b10d-c2fb44987f72","Type":"ContainerDied","Data":"1536ebbdb868face0c9b0cf4008ba1d729d6d89578447d6798ad8fe21700bc31"} Jul 16 17:25:56.875755 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:56.875729 2575 scope.go:117] "RemoveContainer" containerID="1536ebbdb868face0c9b0cf4008ba1d729d6d89578447d6798ad8fe21700bc31" Jul 16 17:25:56.876775 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:56.876749 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:25:56.876879 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:56.876831 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:25:56.876947 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:56.876877 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:25:57.333475 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:57.333390 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:25:57.882017 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:57.881981 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" event={"ID":"d3e80a3e-3027-455e-b10d-c2fb44987f72","Type":"ContainerStarted","Data":"f7722427e28cf1cc19c355603be84a59d850a4aaaf42076c3bf7b9320cb1a3a8"} Jul 16 17:25:57.982145 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:57.982114 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:58.115957 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:58.115932 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:58.829269 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:58.829232 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:58.830812 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:58.830783 2575 patch_prober.go:28] interesting pod/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 16 17:25:58.831020 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:58.830993 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" podUID="d3e80a3e-3027-455e-b10d-c2fb44987f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 16 17:25:59.175790 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:59.175759 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:25:59.806524 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:59.806473 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:25:59.830007 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:59.829962 2575 patch_prober.go:28] interesting pod/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 16 17:25:59.830194 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:25:59.830046 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" podUID="d3e80a3e-3027-455e-b10d-c2fb44987f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 16 17:26:00.829874 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:00.829835 2575 patch_prober.go:28] interesting pod/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 16 17:26:00.830516 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:00.830480 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" podUID="d3e80a3e-3027-455e-b10d-c2fb44987f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 16 17:26:01.830063 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:01.830014 2575 patch_prober.go:28] interesting pod/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 16 17:26:01.830602 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:01.830088 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" podUID="d3e80a3e-3027-455e-b10d-c2fb44987f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 16 17:26:02.833331 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:02.833302 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:26:02.834411 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:02.834392 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:26:02.980635 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:02.980608 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 17:26:02.985348 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:02.985321 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 17:26:05.129409 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.129373 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q"] Jul 16 17:26:05.129811 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.129563 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" podUID="b0a47e04-2b57-441a-90ac-2887631e94e9" containerName="maas-api" containerID="cri-o://d30c78f7088d6fdbad0788f8f0aa1ef2ab7df6c06046272d5ee34400a52fc1c4" gracePeriod=30 Jul 16 17:26:05.189936 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.189912 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:05.379918 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.379852 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" Jul 16 17:26:05.407483 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.407459 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a47e04-2b57-441a-90ac-2887631e94e9-openshift-service-ca\") pod \"b0a47e04-2b57-441a-90ac-2887631e94e9\" (UID: \"b0a47e04-2b57-441a-90ac-2887631e94e9\") " Jul 16 17:26:05.407614 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.407529 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96r7z\" (UniqueName: \"kubernetes.io/projected/b0a47e04-2b57-441a-90ac-2887631e94e9-kube-api-access-96r7z\") pod \"b0a47e04-2b57-441a-90ac-2887631e94e9\" (UID: \"b0a47e04-2b57-441a-90ac-2887631e94e9\") " Jul 16 17:26:05.407614 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.407575 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a47e04-2b57-441a-90ac-2887631e94e9-maas-api-tls\") pod \"b0a47e04-2b57-441a-90ac-2887631e94e9\" (UID: \"b0a47e04-2b57-441a-90ac-2887631e94e9\") " Jul 16 17:26:05.407961 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.407929 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a47e04-2b57-441a-90ac-2887631e94e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0a47e04-2b57-441a-90ac-2887631e94e9" (UID: "b0a47e04-2b57-441a-90ac-2887631e94e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:26:05.409612 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.409593 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a47e04-2b57-441a-90ac-2887631e94e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0a47e04-2b57-441a-90ac-2887631e94e9" (UID: "b0a47e04-2b57-441a-90ac-2887631e94e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:26:05.409679 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.409607 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a47e04-2b57-441a-90ac-2887631e94e9-kube-api-access-96r7z" (OuterVolumeSpecName: "kube-api-access-96r7z") pod "b0a47e04-2b57-441a-90ac-2887631e94e9" (UID: "b0a47e04-2b57-441a-90ac-2887631e94e9"). InnerVolumeSpecName "kube-api-access-96r7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:26:05.508698 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.508666 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96r7z\" (UniqueName: \"kubernetes.io/projected/b0a47e04-2b57-441a-90ac-2887631e94e9-kube-api-access-96r7z\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:26:05.508698 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.508697 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a47e04-2b57-441a-90ac-2887631e94e9-maas-api-tls\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:26:05.508698 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.508706 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a47e04-2b57-441a-90ac-2887631e94e9-openshift-service-ca\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:26:05.926713 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.926679 2575 generic.go:356] "Generic (PLEG): container finished" podID="b0a47e04-2b57-441a-90ac-2887631e94e9" containerID="d30c78f7088d6fdbad0788f8f0aa1ef2ab7df6c06046272d5ee34400a52fc1c4" exitCode=0 Jul 16 17:26:05.927018 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.926721 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" Jul 16 17:26:05.927018 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.926762 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" event={"ID":"b0a47e04-2b57-441a-90ac-2887631e94e9","Type":"ContainerDied","Data":"d30c78f7088d6fdbad0788f8f0aa1ef2ab7df6c06046272d5ee34400a52fc1c4"} Jul 16 17:26:05.927018 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.926818 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q" event={"ID":"b0a47e04-2b57-441a-90ac-2887631e94e9","Type":"ContainerDied","Data":"36955bc0ac0f4b3b56e2aa59812c22be9dbfd40fb435b9f5c764a9a127792938"} Jul 16 17:26:05.927018 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.926840 2575 scope.go:117] "RemoveContainer" containerID="d30c78f7088d6fdbad0788f8f0aa1ef2ab7df6c06046272d5ee34400a52fc1c4" Jul 16 17:26:05.935765 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.935747 2575 scope.go:117] "RemoveContainer" containerID="d30c78f7088d6fdbad0788f8f0aa1ef2ab7df6c06046272d5ee34400a52fc1c4" Jul 16 17:26:05.936003 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:26:05.935985 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30c78f7088d6fdbad0788f8f0aa1ef2ab7df6c06046272d5ee34400a52fc1c4\": container with ID starting with d30c78f7088d6fdbad0788f8f0aa1ef2ab7df6c06046272d5ee34400a52fc1c4 not found: ID does not exist" containerID="d30c78f7088d6fdbad0788f8f0aa1ef2ab7df6c06046272d5ee34400a52fc1c4" Jul 16 17:26:05.936061 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.936014 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30c78f7088d6fdbad0788f8f0aa1ef2ab7df6c06046272d5ee34400a52fc1c4"} err="failed to get container status \"d30c78f7088d6fdbad0788f8f0aa1ef2ab7df6c06046272d5ee34400a52fc1c4\": rpc error: code = NotFound desc = could not find container \"d30c78f7088d6fdbad0788f8f0aa1ef2ab7df6c06046272d5ee34400a52fc1c4\": container with ID starting with d30c78f7088d6fdbad0788f8f0aa1ef2ab7df6c06046272d5ee34400a52fc1c4 not found: ID does not exist" Jul 16 17:26:05.965317 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.965295 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q"] Jul 16 17:26:05.966815 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:05.966798 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-36dc65-58c9f9bc48-szf4q"] Jul 16 17:26:06.908760 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:06.908724 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0a47e04-2b57-441a-90ac-2887631e94e9" path="/var/lib/kubelet/pods/b0a47e04-2b57-441a-90ac-2887631e94e9/volumes" Jul 16 17:26:08.496514 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:08.496482 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:08.661631 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:08.661595 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:26:09.729274 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:09.729239 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:09.869529 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:09.869501 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:17.113185 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:17.113153 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:17.241413 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:17.241375 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:17.594809 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:17.594729 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:17.879811 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:17.879781 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:18.420052 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:18.420010 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:25.568992 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:25.568958 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h"] Jul 16 17:26:25.569463 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:25.569179 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" podUID="20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9" containerName="maas-api" containerID="cri-o://39ec852b4f675c40432f396bf417950e8485b7e6fd8c1a6d1654478a4fe8b282" gracePeriod=30 Jul 16 17:26:25.629349 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:25.629322 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:25.820251 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:25.820197 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" Jul 16 17:26:25.887086 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:25.887055 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9-openshift-service-ca\") pod \"20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9\" (UID: \"20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9\") " Jul 16 17:26:25.887286 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:25.887158 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh2b8\" (UniqueName: \"kubernetes.io/projected/20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9-kube-api-access-bh2b8\") pod \"20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9\" (UID: \"20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9\") " Jul 16 17:26:25.887286 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:25.887197 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9-maas-api-tls\") pod \"20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9\" (UID: \"20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9\") " Jul 16 17:26:25.887486 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:25.887459 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9" (UID: "20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:26:25.889312 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:25.889281 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9" (UID: "20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:26:25.889426 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:25.889321 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9-kube-api-access-bh2b8" (OuterVolumeSpecName: "kube-api-access-bh2b8") pod "20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9" (UID: "20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9"). InnerVolumeSpecName "kube-api-access-bh2b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:26:25.988747 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:25.988716 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9-openshift-service-ca\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:26:25.988747 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:25.988741 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh2b8\" (UniqueName: \"kubernetes.io/projected/20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9-kube-api-access-bh2b8\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:26:25.988747 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:25.988751 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9-maas-api-tls\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:26:26.026831 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:26.026799 2575 generic.go:356] "Generic (PLEG): container finished" podID="20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9" containerID="39ec852b4f675c40432f396bf417950e8485b7e6fd8c1a6d1654478a4fe8b282" exitCode=0 Jul 16 17:26:26.026972 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:26.026848 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" Jul 16 17:26:26.026972 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:26.026891 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" event={"ID":"20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9","Type":"ContainerDied","Data":"39ec852b4f675c40432f396bf417950e8485b7e6fd8c1a6d1654478a4fe8b282"} Jul 16 17:26:26.026972 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:26.026951 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h" event={"ID":"20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9","Type":"ContainerDied","Data":"c1bd8247b86e4061205c61a3df217f0c8e1fb3be9fab44341ba85a0dab92b062"} Jul 16 17:26:26.027172 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:26.026982 2575 scope.go:117] "RemoveContainer" containerID="39ec852b4f675c40432f396bf417950e8485b7e6fd8c1a6d1654478a4fe8b282" Jul 16 17:26:26.036971 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:26.036956 2575 scope.go:117] "RemoveContainer" containerID="39ec852b4f675c40432f396bf417950e8485b7e6fd8c1a6d1654478a4fe8b282" Jul 16 17:26:26.037247 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:26:26.037226 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39ec852b4f675c40432f396bf417950e8485b7e6fd8c1a6d1654478a4fe8b282\": container with ID starting with 39ec852b4f675c40432f396bf417950e8485b7e6fd8c1a6d1654478a4fe8b282 not found: ID does not exist" containerID="39ec852b4f675c40432f396bf417950e8485b7e6fd8c1a6d1654478a4fe8b282" Jul 16 17:26:26.037317 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:26.037254 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ec852b4f675c40432f396bf417950e8485b7e6fd8c1a6d1654478a4fe8b282"} err="failed to get container status \"39ec852b4f675c40432f396bf417950e8485b7e6fd8c1a6d1654478a4fe8b282\": rpc error: code = NotFound desc = could not find container \"39ec852b4f675c40432f396bf417950e8485b7e6fd8c1a6d1654478a4fe8b282\": container with ID starting with 39ec852b4f675c40432f396bf417950e8485b7e6fd8c1a6d1654478a4fe8b282 not found: ID does not exist" Jul 16 17:26:26.067151 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:26.067121 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h"] Jul 16 17:26:26.074161 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:26.074083 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7ca5c1-84489658f8-dgw8h"] Jul 16 17:26:26.909064 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:26.909034 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9" path="/var/lib/kubelet/pods/20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9/volumes" Jul 16 17:26:26.948188 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:26.948156 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:29.284305 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:29.284272 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:26:34.488558 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:34.488523 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:34.633754 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:34.633720 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:34.831272 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:34.831193 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s"] Jul 16 17:26:34.831424 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:34.831402 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" podUID="d3e80a3e-3027-455e-b10d-c2fb44987f72" containerName="istio-proxy" containerID="cri-o://f7722427e28cf1cc19c355603be84a59d850a4aaaf42076c3bf7b9320cb1a3a8" gracePeriod=30 Jul 16 17:26:34.874202 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:34.874176 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:35.153543 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:35.153512 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:35.529144 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:35.529038 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:35.985043 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:35.985013 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:36.376616 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:36.376575 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:38.210148 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.210090 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:38.361452 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.361422 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:38.388087 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.388059 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm"] Jul 16 17:26:38.388387 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.388374 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9" containerName="maas-api" Jul 16 17:26:38.388387 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.388387 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9" containerName="maas-api" Jul 16 17:26:38.388500 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.388407 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a47e04-2b57-441a-90ac-2887631e94e9" containerName="maas-api" Jul 16 17:26:38.388500 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.388413 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a47e04-2b57-441a-90ac-2887631e94e9" containerName="maas-api" Jul 16 17:26:38.388500 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.388483 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0a47e04-2b57-441a-90ac-2887631e94e9" containerName="maas-api" Jul 16 17:26:38.388500 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.388492 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="20f73e9d-9fb6-4f28-a13d-8b9f2678c7d9" containerName="maas-api" Jul 16 17:26:38.391297 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.391281 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" Jul 16 17:26:38.394337 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.394318 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-b44c0f-serving-cert\"" Jul 16 17:26:38.394430 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.394338 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 17:26:38.395517 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.395497 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bff62\"" Jul 16 17:26:38.396646 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.396630 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 17:26:38.428634 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.428599 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm"] Jul 16 17:26:38.504158 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.504052 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a55619-2afc-4257-b199-c732a5dc658b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm\" (UID: \"b2a55619-2afc-4257-b199-c732a5dc658b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" Jul 16 17:26:38.504158 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.504086 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbtkv\" (UniqueName: \"kubernetes.io/projected/b2a55619-2afc-4257-b199-c732a5dc658b-kube-api-access-rbtkv\") pod \"maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm\" (UID: \"b2a55619-2afc-4257-b199-c732a5dc658b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" Jul 16 17:26:38.504352 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.504246 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a55619-2afc-4257-b199-c732a5dc658b-maas-api-tls\") pod \"maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm\" (UID: \"b2a55619-2afc-4257-b199-c732a5dc658b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" Jul 16 17:26:38.521291 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.521266 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:38.605548 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.605510 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a55619-2afc-4257-b199-c732a5dc658b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm\" (UID: \"b2a55619-2afc-4257-b199-c732a5dc658b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" Jul 16 17:26:38.605548 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.605548 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbtkv\" (UniqueName: \"kubernetes.io/projected/b2a55619-2afc-4257-b199-c732a5dc658b-kube-api-access-rbtkv\") pod \"maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm\" (UID: \"b2a55619-2afc-4257-b199-c732a5dc658b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" Jul 16 17:26:38.605771 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.605632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a55619-2afc-4257-b199-c732a5dc658b-maas-api-tls\") pod \"maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm\" (UID: \"b2a55619-2afc-4257-b199-c732a5dc658b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" Jul 16 17:26:38.606169 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.606149 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a55619-2afc-4257-b199-c732a5dc658b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm\" (UID: \"b2a55619-2afc-4257-b199-c732a5dc658b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" Jul 16 17:26:38.607989 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.607966 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a55619-2afc-4257-b199-c732a5dc658b-maas-api-tls\") pod \"maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm\" (UID: \"b2a55619-2afc-4257-b199-c732a5dc658b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" Jul 16 17:26:38.637577 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.637542 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbtkv\" (UniqueName: \"kubernetes.io/projected/b2a55619-2afc-4257-b199-c732a5dc658b-kube-api-access-rbtkv\") pod \"maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm\" (UID: \"b2a55619-2afc-4257-b199-c732a5dc658b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" Jul 16 17:26:38.673029 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.672996 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:38.702628 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.702602 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" Jul 16 17:26:38.848199 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:38.848169 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm"] Jul 16 17:26:38.851064 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:26:38.851026 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a55619_2afc_4257_b199_c732a5dc658b.slice/crio-3ad3dbfbcf992279318eb766cd183dbc72b38397712a2464fafb3ffe3b20078a WatchSource:0}: Error finding container 3ad3dbfbcf992279318eb766cd183dbc72b38397712a2464fafb3ffe3b20078a: Status 404 returned error can't find the container with id 3ad3dbfbcf992279318eb766cd183dbc72b38397712a2464fafb3ffe3b20078a Jul 16 17:26:39.085370 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:39.085289 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" event={"ID":"b2a55619-2afc-4257-b199-c732a5dc658b","Type":"ContainerStarted","Data":"3ad3dbfbcf992279318eb766cd183dbc72b38397712a2464fafb3ffe3b20078a"} Jul 16 17:26:39.708593 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:39.708558 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:39.864205 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:39.864177 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:40.092158 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.092079 2575 generic.go:356] "Generic (PLEG): container finished" podID="d3e80a3e-3027-455e-b10d-c2fb44987f72" containerID="f7722427e28cf1cc19c355603be84a59d850a4aaaf42076c3bf7b9320cb1a3a8" exitCode=0 Jul 16 17:26:40.092276 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.092127 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" event={"ID":"d3e80a3e-3027-455e-b10d-c2fb44987f72","Type":"ContainerDied","Data":"f7722427e28cf1cc19c355603be84a59d850a4aaaf42076c3bf7b9320cb1a3a8"} Jul 16 17:26:40.092336 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.092274 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" event={"ID":"d3e80a3e-3027-455e-b10d-c2fb44987f72","Type":"ContainerDied","Data":"d57b73385070f735c801f24ce11c3249f7877bd261d8f3b7d3bad8d6e24f971f"} Jul 16 17:26:40.092336 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.092302 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d57b73385070f735c801f24ce11c3249f7877bd261d8f3b7d3bad8d6e24f971f" Jul 16 17:26:40.092336 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.092321 2575 scope.go:117] "RemoveContainer" containerID="1536ebbdb868face0c9b0cf4008ba1d729d6d89578447d6798ad8fe21700bc31" Jul 16 17:26:40.093477 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.093461 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:26:40.093716 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.093689 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" event={"ID":"b2a55619-2afc-4257-b199-c732a5dc658b","Type":"ContainerStarted","Data":"0bda3dd4caecd91b009836dc846c97c6ebfa769567068d1a60cd431c084e8a90"} Jul 16 17:26:40.093828 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.093754 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" Jul 16 17:26:40.185189 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.185142 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" podStartSLOduration=1.751121736 podStartE2EDuration="2.18512707s" totalImagesPullingTime="434.005334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:26:40.184632707 +0000 UTC m=+2137.703357942" watchObservedRunningTime="2026-07-16 17:26:40.18512707 +0000 UTC m=+2137.703852305" Jul 16 17:26:40.219829 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.219805 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-workload-certs\") pod \"d3e80a3e-3027-455e-b10d-c2fb44987f72\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " Jul 16 17:26:40.219963 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.219842 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-token\") pod \"d3e80a3e-3027-455e-b10d-c2fb44987f72\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " Jul 16 17:26:40.219963 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.219877 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-envoy\") pod \"d3e80a3e-3027-455e-b10d-c2fb44987f72\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " Jul 16 17:26:40.220050 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.219981 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6qwc\" (UniqueName: \"kubernetes.io/projected/d3e80a3e-3027-455e-b10d-c2fb44987f72-kube-api-access-s6qwc\") pod \"d3e80a3e-3027-455e-b10d-c2fb44987f72\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " Jul 16 17:26:40.220050 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.220021 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e80a3e-3027-455e-b10d-c2fb44987f72-istiod-ca-cert\") pod \"d3e80a3e-3027-455e-b10d-c2fb44987f72\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " Jul 16 17:26:40.220050 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.220044 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-podinfo\") pod \"d3e80a3e-3027-455e-b10d-c2fb44987f72\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " Jul 16 17:26:40.220247 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.220135 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-data\") pod \"d3e80a3e-3027-455e-b10d-c2fb44987f72\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " Jul 16 17:26:40.220247 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.220146 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d3e80a3e-3027-455e-b10d-c2fb44987f72" (UID: "d3e80a3e-3027-455e-b10d-c2fb44987f72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:26:40.220247 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.220182 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-credential-socket\") pod \"d3e80a3e-3027-455e-b10d-c2fb44987f72\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " Jul 16 17:26:40.220247 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.220234 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-workload-socket\") pod \"d3e80a3e-3027-455e-b10d-c2fb44987f72\" (UID: \"d3e80a3e-3027-455e-b10d-c2fb44987f72\") " Jul 16 17:26:40.220470 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.220454 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e80a3e-3027-455e-b10d-c2fb44987f72-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d3e80a3e-3027-455e-b10d-c2fb44987f72" (UID: "d3e80a3e-3027-455e-b10d-c2fb44987f72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:26:40.220830 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.220573 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e80a3e-3027-455e-b10d-c2fb44987f72-istiod-ca-cert\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:26:40.220830 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.220594 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-workload-certs\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:26:40.220830 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.220777 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d3e80a3e-3027-455e-b10d-c2fb44987f72" (UID: "d3e80a3e-3027-455e-b10d-c2fb44987f72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:26:40.221041 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.220840 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d3e80a3e-3027-455e-b10d-c2fb44987f72" (UID: "d3e80a3e-3027-455e-b10d-c2fb44987f72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:26:40.221480 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.221453 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-data" (OuterVolumeSpecName: "istio-data") pod "d3e80a3e-3027-455e-b10d-c2fb44987f72" (UID: "d3e80a3e-3027-455e-b10d-c2fb44987f72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:26:40.222305 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.222282 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d3e80a3e-3027-455e-b10d-c2fb44987f72" (UID: "d3e80a3e-3027-455e-b10d-c2fb44987f72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:26:40.222411 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.222347 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e80a3e-3027-455e-b10d-c2fb44987f72-kube-api-access-s6qwc" (OuterVolumeSpecName: "kube-api-access-s6qwc") pod "d3e80a3e-3027-455e-b10d-c2fb44987f72" (UID: "d3e80a3e-3027-455e-b10d-c2fb44987f72"). InnerVolumeSpecName "kube-api-access-s6qwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:26:40.222588 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.222570 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d3e80a3e-3027-455e-b10d-c2fb44987f72" (UID: "d3e80a3e-3027-455e-b10d-c2fb44987f72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 17:26:40.222642 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.222599 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-token" (OuterVolumeSpecName: "istio-token") pod "d3e80a3e-3027-455e-b10d-c2fb44987f72" (UID: "d3e80a3e-3027-455e-b10d-c2fb44987f72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:26:40.321419 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.321350 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-podinfo\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:26:40.321419 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.321375 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-data\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:26:40.321419 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.321384 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-credential-socket\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:26:40.321419 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.321393 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-workload-socket\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:26:40.321419 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.321401 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-token\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:26:40.321419 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.321409 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e80a3e-3027-455e-b10d-c2fb44987f72-istio-envoy\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:26:40.321419 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:40.321416 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6qwc\" (UniqueName: \"kubernetes.io/projected/d3e80a3e-3027-455e-b10d-c2fb44987f72-kube-api-access-s6qwc\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:26:41.104701 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:41.104670 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s" Jul 16 17:26:41.176717 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:41.176629 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s"] Jul 16 17:26:41.180136 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:41.180113 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7ca5c1-openshift-default-54678654bf-wtd5s"] Jul 16 17:26:42.909649 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:42.909615 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3e80a3e-3027-455e-b10d-c2fb44987f72" path="/var/lib/kubelet/pods/d3e80a3e-3027-455e-b10d-c2fb44987f72/volumes" Jul 16 17:26:43.682360 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:43.682328 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:44.093880 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.093799 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:44.553065 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.553030 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:26:44.571235 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.571204 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn"] Jul 16 17:26:44.571577 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.571564 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3e80a3e-3027-455e-b10d-c2fb44987f72" containerName="istio-proxy" Jul 16 17:26:44.571638 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.571578 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3e80a3e-3027-455e-b10d-c2fb44987f72" containerName="istio-proxy" Jul 16 17:26:44.571638 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.571587 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3e80a3e-3027-455e-b10d-c2fb44987f72" containerName="istio-proxy" Jul 16 17:26:44.571638 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.571596 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3e80a3e-3027-455e-b10d-c2fb44987f72" containerName="istio-proxy" Jul 16 17:26:44.571790 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.571654 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3e80a3e-3027-455e-b10d-c2fb44987f72" containerName="istio-proxy" Jul 16 17:26:44.576142 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.576122 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.581609 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.581589 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-8d3a22-openshift-default-dockercfg-z746f\"" Jul 16 17:26:44.652553 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.652520 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-workload-certs\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.652743 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.652565 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-workload-socket\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.652743 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.652627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-credential-socket\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.652743 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.652653 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-envoy\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.652743 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.652677 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d66x8\" (UniqueName: \"kubernetes.io/projected/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-kube-api-access-d66x8\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.652953 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.652761 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-token\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.652953 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.652842 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-data\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.652953 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.652872 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istiod-ca-cert\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.652953 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.652921 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-podinfo\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.657720 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.657697 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:44.691629 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.691600 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn"] Jul 16 17:26:44.753406 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.753381 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-data\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.753557 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.753424 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istiod-ca-cert\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.753557 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.753450 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-podinfo\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.753557 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.753473 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-workload-certs\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.753557 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.753496 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-workload-socket\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.753557 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.753537 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-credential-socket\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.753557 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.753553 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-envoy\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.753830 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.753576 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d66x8\" (UniqueName: \"kubernetes.io/projected/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-kube-api-access-d66x8\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.753830 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.753610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-token\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.753936 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.753864 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-data\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.753992 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.753928 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-credential-socket\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.754082 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.754060 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-workload-certs\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.754338 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.754316 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-workload-socket\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.754444 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.754427 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istiod-ca-cert\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.755859 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.755836 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-envoy\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.756207 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.756178 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-podinfo\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.766835 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.766803 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-token\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.767006 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.766898 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d66x8\" (UniqueName: \"kubernetes.io/projected/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-kube-api-access-d66x8\") pod \"e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.886433 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.886403 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:44.927741 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:44.927715 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:45.042228 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:45.042197 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn"] Jul 16 17:26:45.044046 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:26:45.044020 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b29a1a_5df3_4ccd_8214_e84dc2e857fa.slice/crio-e3aeca43d7b073a39931eab06da2b339e05a8797517f741602e5c59ded9b98d7 WatchSource:0}: Error finding container e3aeca43d7b073a39931eab06da2b339e05a8797517f741602e5c59ded9b98d7: Status 404 returned error can't find the container with id e3aeca43d7b073a39931eab06da2b339e05a8797517f741602e5c59ded9b98d7 Jul 16 17:26:45.046057 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:45.046026 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:26:45.046237 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:45.046139 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:26:45.046237 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:45.046225 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:26:45.124646 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:45.124607 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" event={"ID":"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa","Type":"ContainerStarted","Data":"8b84fb835f33157d970f055d1639b6af93e136d414b9bcff8e35d287fff1eaa6"} Jul 16 17:26:45.124935 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:45.124677 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" event={"ID":"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa","Type":"ContainerStarted","Data":"e3aeca43d7b073a39931eab06da2b339e05a8797517f741602e5c59ded9b98d7"} Jul 16 17:26:45.163932 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:45.163830 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" podStartSLOduration=1.163808538 podStartE2EDuration="1.163808538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:26:45.16144701 +0000 UTC m=+2142.680172265" watchObservedRunningTime="2026-07-16 17:26:45.163808538 +0000 UTC m=+2142.682533773" Jul 16 17:26:45.887349 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:45.887318 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:46.109973 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:46.109941 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" Jul 16 17:26:46.323460 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:46.323376 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:46.457409 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:46.457371 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:46.724067 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:46.724039 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:46.733130 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:46.730609 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6"] Jul 16 17:26:46.733130 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:46.731623 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3e80a3e-3027-455e-b10d-c2fb44987f72" containerName="istio-proxy" Jul 16 17:26:46.737915 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:46.737886 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" Jul 16 17:26:46.748653 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:46.748633 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-8d3a22-serving-cert\"" Jul 16 17:26:46.767166 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:46.767144 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b227b6-56ee-4ae3-927c-0f62ca637f37-maas-api-tls\") pod \"maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6\" (UID: \"a2b227b6-56ee-4ae3-927c-0f62ca637f37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" Jul 16 17:26:46.767263 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:46.767197 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b227b6-56ee-4ae3-927c-0f62ca637f37-openshift-service-ca\") pod \"maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6\" (UID: \"a2b227b6-56ee-4ae3-927c-0f62ca637f37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" Jul 16 17:26:46.767263 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:46.767223 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl55l\" (UniqueName: \"kubernetes.io/projected/a2b227b6-56ee-4ae3-927c-0f62ca637f37-kube-api-access-tl55l\") pod \"maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6\" (UID: \"a2b227b6-56ee-4ae3-927c-0f62ca637f37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" Jul 16 17:26:46.778887 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:46.778864 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6"] Jul 16 17:26:46.867750 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:46.867706 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b227b6-56ee-4ae3-927c-0f62ca637f37-maas-api-tls\") pod \"maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6\" (UID: \"a2b227b6-56ee-4ae3-927c-0f62ca637f37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" Jul 16 17:26:46.867915 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:46.867804 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b227b6-56ee-4ae3-927c-0f62ca637f37-openshift-service-ca\") pod \"maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6\" (UID: \"a2b227b6-56ee-4ae3-927c-0f62ca637f37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" Jul 16 17:26:46.867915 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:46.867837 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl55l\" (UniqueName: \"kubernetes.io/projected/a2b227b6-56ee-4ae3-927c-0f62ca637f37-kube-api-access-tl55l\") pod \"maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6\" (UID: \"a2b227b6-56ee-4ae3-927c-0f62ca637f37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" Jul 16 17:26:46.868522 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:46.868504 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b227b6-56ee-4ae3-927c-0f62ca637f37-openshift-service-ca\") pod \"maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6\" (UID: \"a2b227b6-56ee-4ae3-927c-0f62ca637f37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" Jul 16 17:26:46.870272 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:46.870248 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b227b6-56ee-4ae3-927c-0f62ca637f37-maas-api-tls\") pod \"maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6\" (UID: \"a2b227b6-56ee-4ae3-927c-0f62ca637f37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" Jul 16 17:26:46.884811 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:46.884789 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl55l\" (UniqueName: \"kubernetes.io/projected/a2b227b6-56ee-4ae3-927c-0f62ca637f37-kube-api-access-tl55l\") pod \"maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6\" (UID: \"a2b227b6-56ee-4ae3-927c-0f62ca637f37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" Jul 16 17:26:46.891916 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:46.891897 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:47.051761 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:47.051676 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" Jul 16 17:26:47.123510 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:47.123479 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:47.133937 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:47.133914 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:47.135366 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:47.134925 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:26:47.237681 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:47.237654 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6"] Jul 16 17:26:47.240588 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:26:47.240556 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b227b6_56ee_4ae3_927c_0f62ca637f37.slice/crio-040efc697ba7ee9450634c0d58a55f0818fc00a28ca3fbb8812a7fd443dd71e6 WatchSource:0}: Error finding container 040efc697ba7ee9450634c0d58a55f0818fc00a28ca3fbb8812a7fd443dd71e6: Status 404 returned error can't find the container with id 040efc697ba7ee9450634c0d58a55f0818fc00a28ca3fbb8812a7fd443dd71e6 Jul 16 17:26:48.139946 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:48.139912 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" event={"ID":"a2b227b6-56ee-4ae3-927c-0f62ca637f37","Type":"ContainerStarted","Data":"6f247d609f679daf87347213b783a1f24865dded808bf6f5c1e06b2eedc66c1b"} Jul 16 17:26:48.140395 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:48.139964 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" event={"ID":"a2b227b6-56ee-4ae3-927c-0f62ca637f37","Type":"ContainerStarted","Data":"040efc697ba7ee9450634c0d58a55f0818fc00a28ca3fbb8812a7fd443dd71e6"} Jul 16 17:26:48.140395 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:48.140130 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" Jul 16 17:26:48.176936 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:48.176888 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" podStartSLOduration=1.869761005 podStartE2EDuration="2.176874069s" totalImagesPullingTime="307.113064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:26:48.175520403 +0000 UTC m=+2145.694245637" watchObservedRunningTime="2026-07-16 17:26:48.176874069 +0000 UTC m=+2145.695599302" Jul 16 17:26:49.013129 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:49.013081 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:49.161517 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:49.161488 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:51.801957 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:51.801918 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:26:53.499375 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:53.499340 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:26:54.151438 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:26:54.151405 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" Jul 16 17:27:00.680196 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:00.680164 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:27:01.449048 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:01.449014 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:27:01.548876 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:01.548845 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:27:02.135653 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:02.135620 2575 patch_prober.go:28] interesting pod/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 16 17:27:02.136028 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:02.135687 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" podUID="e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 16 17:27:02.219561 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:02.219532 2575 generic.go:356] "Generic (PLEG): container finished" podID="e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" containerID="8b84fb835f33157d970f055d1639b6af93e136d414b9bcff8e35d287fff1eaa6" exitCode=0 Jul 16 17:27:02.219734 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:02.219611 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" event={"ID":"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa","Type":"ContainerDied","Data":"8b84fb835f33157d970f055d1639b6af93e136d414b9bcff8e35d287fff1eaa6"} Jul 16 17:27:02.220001 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:02.219988 2575 scope.go:117] "RemoveContainer" containerID="8b84fb835f33157d970f055d1639b6af93e136d414b9bcff8e35d287fff1eaa6" Jul 16 17:27:02.221022 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:02.221000 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:27:02.221078 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:02.221068 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:27:02.221135 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:02.221115 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:27:03.228365 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:03.228320 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" event={"ID":"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa","Type":"ContainerStarted","Data":"e8daf8cda435b44194e3f8e82019f1f9ec4546cf57e80ee005eb429ef3a40732"} Jul 16 17:27:03.887249 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:03.887208 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:27:03.887582 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:03.887543 2575 patch_prober.go:28] interesting pod/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 16 17:27:03.887698 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:03.887610 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" podUID="e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 16 17:27:04.271458 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:04.271378 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:27:04.887304 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:04.887267 2575 patch_prober.go:28] interesting pod/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 16 17:27:04.887508 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:04.887322 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" podUID="e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 16 17:27:05.886817 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:05.886779 2575 patch_prober.go:28] interesting pod/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 16 17:27:05.887236 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:05.886847 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" podUID="e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 16 17:27:06.546751 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:06.546720 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:27:06.887245 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:06.887213 2575 patch_prober.go:28] interesting pod/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 16 17:27:06.887659 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:06.887277 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" podUID="e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 16 17:27:07.890773 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:07.890746 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:27:07.891258 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:07.891075 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:27:07.891788 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:07.891769 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:27:10.312771 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:10.312708 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:27:10.409735 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:10.409701 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:27:12.810337 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:12.810306 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:27:46.639286 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:46.639251 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:27:50.374336 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:27:50.374302 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:28:13.977178 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:28:13.977139 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:28:21.569167 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:28:21.569117 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:28:21.658335 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:28:21.658304 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:28:30.797070 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:28:30.796987 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:28:44.794169 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:28:44.794136 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:29:03.297713 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:29:03.297676 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:29:16.897616 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:29:16.897583 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:29:34.897329 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:29:34.897292 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:29:52.880211 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:29:52.880178 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:29:57.985939 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:29:57.985744 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:30:00.150553 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:00.150518 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737050-8s8pn"] Jul 16 17:30:00.154071 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:00.154048 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b44c0f-29737050-z67jc"] Jul 16 17:30:00.154240 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:00.154195 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737050-8s8pn" Jul 16 17:30:00.157691 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:00.157674 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b44c0f-29737050-z67jc" Jul 16 17:30:00.159748 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:00.159722 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t6smk\"" Jul 16 17:30:00.165474 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:00.165455 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737050-8s8pn" Jul 16 17:30:00.166394 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:00.166372 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737050-8s8pn"] Jul 16 17:30:00.170501 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:00.170432 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b44c0f-29737050-z67jc"] Jul 16 17:30:00.176556 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:00.176535 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b44c0f-29737050-z67jc" Jul 16 17:30:00.323458 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:00.323434 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737050-8s8pn"] Jul 16 17:30:00.325170 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:30:00.325089 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1774ba3b_813c_4238_bc50_51455341a5c1.slice/crio-d31bda647f58ba35315e06783162e9693dd54ebff534661efebfa51fbd16beb6 WatchSource:0}: Error finding container d31bda647f58ba35315e06783162e9693dd54ebff534661efebfa51fbd16beb6: Status 404 returned error can't find the container with id d31bda647f58ba35315e06783162e9693dd54ebff534661efebfa51fbd16beb6 Jul 16 17:30:00.554596 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:00.554574 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b44c0f-29737050-z67jc"] Jul 16 17:30:00.556023 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:30:00.555995 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aeae13c_c29e_4728_a7e7_2a258553f0e8.slice/crio-633a4e7414d52b415c1a74618fedbf2b4090f3f09701ca4557e1390fcf179b72 WatchSource:0}: Error finding container 633a4e7414d52b415c1a74618fedbf2b4090f3f09701ca4557e1390fcf179b72: Status 404 returned error can't find the container with id 633a4e7414d52b415c1a74618fedbf2b4090f3f09701ca4557e1390fcf179b72 Jul 16 17:30:01.057931 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:01.057893 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737050-8s8pn" event={"ID":"1774ba3b-813c-4238-bc50-51455341a5c1","Type":"ContainerStarted","Data":"260378cc080eebaf437753493a2997f26567ac77ae9a451f756b4abbe8d4818a"} Jul 16 17:30:01.058151 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:01.057957 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737050-8s8pn" event={"ID":"1774ba3b-813c-4238-bc50-51455341a5c1","Type":"ContainerStarted","Data":"d31bda647f58ba35315e06783162e9693dd54ebff534661efebfa51fbd16beb6"} Jul 16 17:30:01.059382 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:01.059359 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b44c0f-29737050-z67jc" event={"ID":"3aeae13c-c29e-4728-a7e7-2a258553f0e8","Type":"ContainerStarted","Data":"667e7cc166be92295f666f170e3246556c6454e67f029513cba8aa06898d78d7"} Jul 16 17:30:01.059476 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:01.059397 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b44c0f-29737050-z67jc" event={"ID":"3aeae13c-c29e-4728-a7e7-2a258553f0e8","Type":"ContainerStarted","Data":"633a4e7414d52b415c1a74618fedbf2b4090f3f09701ca4557e1390fcf179b72"} Jul 16 17:30:01.086583 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:01.086536 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737050-8s8pn" podStartSLOduration=1.08652301 podStartE2EDuration="1.08652301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:01.084688172 +0000 UTC m=+2338.603413407" watchObservedRunningTime="2026-07-16 17:30:01.08652301 +0000 UTC m=+2338.605248241" Jul 16 17:30:01.120880 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:01.120837 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b44c0f-29737050-z67jc" podStartSLOduration=1.120822803 podStartE2EDuration="1.120822803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:30:01.118352768 +0000 UTC m=+2338.637078002" watchObservedRunningTime="2026-07-16 17:30:01.120822803 +0000 UTC m=+2338.639548014" Jul 16 17:30:01.143316 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:01.143294 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:30:02.065423 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:02.065390 2575 generic.go:356] "Generic (PLEG): container finished" podID="1774ba3b-813c-4238-bc50-51455341a5c1" containerID="260378cc080eebaf437753493a2997f26567ac77ae9a451f756b4abbe8d4818a" exitCode=0 Jul 16 17:30:02.065877 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:02.065440 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737050-8s8pn" event={"ID":"1774ba3b-813c-4238-bc50-51455341a5c1","Type":"ContainerDied","Data":"260378cc080eebaf437753493a2997f26567ac77ae9a451f756b4abbe8d4818a"} Jul 16 17:30:02.066989 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:02.066967 2575 generic.go:356] "Generic (PLEG): container finished" podID="3aeae13c-c29e-4728-a7e7-2a258553f0e8" containerID="667e7cc166be92295f666f170e3246556c6454e67f029513cba8aa06898d78d7" exitCode=0 Jul 16 17:30:02.067108 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:02.066994 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b44c0f-29737050-z67jc" event={"ID":"3aeae13c-c29e-4728-a7e7-2a258553f0e8","Type":"ContainerDied","Data":"667e7cc166be92295f666f170e3246556c6454e67f029513cba8aa06898d78d7"} Jul 16 17:30:03.222499 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:03.222476 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b44c0f-29737050-z67jc" Jul 16 17:30:03.225200 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:03.225181 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737050-8s8pn" Jul 16 17:30:04.078080 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:04.078042 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b44c0f-29737050-z67jc" Jul 16 17:30:04.078297 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:04.078072 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b44c0f-29737050-z67jc" event={"ID":"3aeae13c-c29e-4728-a7e7-2a258553f0e8","Type":"ContainerDied","Data":"633a4e7414d52b415c1a74618fedbf2b4090f3f09701ca4557e1390fcf179b72"} Jul 16 17:30:04.078297 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:04.078143 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="633a4e7414d52b415c1a74618fedbf2b4090f3f09701ca4557e1390fcf179b72" Jul 16 17:30:04.079366 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:04.079347 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737050-8s8pn" event={"ID":"1774ba3b-813c-4238-bc50-51455341a5c1","Type":"ContainerDied","Data":"d31bda647f58ba35315e06783162e9693dd54ebff534661efebfa51fbd16beb6"} Jul 16 17:30:04.079466 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:04.079379 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d31bda647f58ba35315e06783162e9693dd54ebff534661efebfa51fbd16beb6" Jul 16 17:30:04.079466 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:04.079380 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737050-8s8pn" Jul 16 17:30:09.888742 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:09.888707 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:30:15.612446 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:15.612420 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:30:17.263900 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:17.263862 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:30:21.090958 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:21.090916 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:30:21.182146 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:21.182088 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:30:22.783743 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:22.783712 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:30:23.386605 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:23.386575 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:30:26.266027 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:26.265992 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:30:38.166131 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:38.166076 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:30:42.184519 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:30:42.184484 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:31:03.008229 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:31:03.008199 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 17:31:03.013631 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:31:03.013612 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fj6nb_6ed467d2-8e31-4fc1-8631-3972722764e6/ovn-acl-logging/0.log" Jul 16 17:31:21.976767 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:31:21.976730 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:31:37.874686 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:31:37.874604 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:31:44.821839 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:31:44.821808 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:31:45.669133 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:31:45.669071 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:31:49.163746 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:31:49.163695 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:31:49.237670 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:31:49.237640 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:31:50.523390 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:31:50.523354 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:31:55.973949 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:31:55.973903 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:32:02.078529 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:32:02.078495 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:32:02.183499 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:32:02.183470 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:32:03.098718 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:32:03.098688 2575 scope.go:117] "RemoveContainer" containerID="f7722427e28cf1cc19c355603be84a59d850a4aaaf42076c3bf7b9320cb1a3a8" Jul 16 17:32:11.247174 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:32:11.247136 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:32:11.344809 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:32:11.344779 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:32:17.666844 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:32:17.666808 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:32:22.112662 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:32:22.112622 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:32:42.568547 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:32:42.568509 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:32:57.573583 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:32:57.573495 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:33:04.583554 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:04.583515 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:33:05.217343 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:05.217310 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:33:10.796412 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:10.796373 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:33:25.639883 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:25.639848 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:33:32.154531 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:32.154493 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm"] Jul 16 17:33:32.154972 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:32.154794 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" podUID="b2a55619-2afc-4257-b199-c732a5dc658b" containerName="maas-api" containerID="cri-o://0bda3dd4caecd91b009836dc846c97c6ebfa769567068d1a60cd431c084e8a90" gracePeriod=30 Jul 16 17:33:32.236122 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:32.236022 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:33:32.258133 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:32.258102 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b44c0f-29737050-z67jc"] Jul 16 17:33:32.261515 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:32.261493 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b44c0f-29737050-z67jc"] Jul 16 17:33:32.403388 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:32.403361 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" Jul 16 17:33:32.497958 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:32.497873 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a55619-2afc-4257-b199-c732a5dc658b-maas-api-tls\") pod \"b2a55619-2afc-4257-b199-c732a5dc658b\" (UID: \"b2a55619-2afc-4257-b199-c732a5dc658b\") " Jul 16 17:33:32.497958 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:32.497935 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a55619-2afc-4257-b199-c732a5dc658b-openshift-service-ca\") pod \"b2a55619-2afc-4257-b199-c732a5dc658b\" (UID: \"b2a55619-2afc-4257-b199-c732a5dc658b\") " Jul 16 17:33:32.498254 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:32.497986 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbtkv\" (UniqueName: \"kubernetes.io/projected/b2a55619-2afc-4257-b199-c732a5dc658b-kube-api-access-rbtkv\") pod \"b2a55619-2afc-4257-b199-c732a5dc658b\" (UID: \"b2a55619-2afc-4257-b199-c732a5dc658b\") " Jul 16 17:33:32.498395 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:32.498353 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a55619-2afc-4257-b199-c732a5dc658b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b2a55619-2afc-4257-b199-c732a5dc658b" (UID: "b2a55619-2afc-4257-b199-c732a5dc658b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:33:32.500049 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:32.500028 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a55619-2afc-4257-b199-c732a5dc658b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b2a55619-2afc-4257-b199-c732a5dc658b" (UID: "b2a55619-2afc-4257-b199-c732a5dc658b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:33:32.500155 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:32.500074 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a55619-2afc-4257-b199-c732a5dc658b-kube-api-access-rbtkv" (OuterVolumeSpecName: "kube-api-access-rbtkv") pod "b2a55619-2afc-4257-b199-c732a5dc658b" (UID: "b2a55619-2afc-4257-b199-c732a5dc658b"). InnerVolumeSpecName "kube-api-access-rbtkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:33:32.598659 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:32.598624 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2a55619-2afc-4257-b199-c732a5dc658b-maas-api-tls\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:33:32.598659 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:32.598649 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a55619-2afc-4257-b199-c732a5dc658b-openshift-service-ca\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:33:32.598659 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:32.598660 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbtkv\" (UniqueName: \"kubernetes.io/projected/b2a55619-2afc-4257-b199-c732a5dc658b-kube-api-access-rbtkv\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:33:32.911196 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:32.911165 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3aeae13c-c29e-4728-a7e7-2a258553f0e8" path="/var/lib/kubelet/pods/3aeae13c-c29e-4728-a7e7-2a258553f0e8/volumes" Jul 16 17:33:33.054612 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:33.054581 2575 generic.go:356] "Generic (PLEG): container finished" podID="b2a55619-2afc-4257-b199-c732a5dc658b" containerID="0bda3dd4caecd91b009836dc846c97c6ebfa769567068d1a60cd431c084e8a90" exitCode=0 Jul 16 17:33:33.054786 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:33.054626 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" Jul 16 17:33:33.054786 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:33.054641 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" event={"ID":"b2a55619-2afc-4257-b199-c732a5dc658b","Type":"ContainerDied","Data":"0bda3dd4caecd91b009836dc846c97c6ebfa769567068d1a60cd431c084e8a90"} Jul 16 17:33:33.054786 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:33.054682 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm" event={"ID":"b2a55619-2afc-4257-b199-c732a5dc658b","Type":"ContainerDied","Data":"3ad3dbfbcf992279318eb766cd183dbc72b38397712a2464fafb3ffe3b20078a"} Jul 16 17:33:33.054786 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:33.054702 2575 scope.go:117] "RemoveContainer" containerID="0bda3dd4caecd91b009836dc846c97c6ebfa769567068d1a60cd431c084e8a90" Jul 16 17:33:33.063524 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:33.063503 2575 scope.go:117] "RemoveContainer" containerID="0bda3dd4caecd91b009836dc846c97c6ebfa769567068d1a60cd431c084e8a90" Jul 16 17:33:33.063775 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:33:33.063754 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bda3dd4caecd91b009836dc846c97c6ebfa769567068d1a60cd431c084e8a90\": container with ID starting with 0bda3dd4caecd91b009836dc846c97c6ebfa769567068d1a60cd431c084e8a90 not found: ID does not exist" containerID="0bda3dd4caecd91b009836dc846c97c6ebfa769567068d1a60cd431c084e8a90" Jul 16 17:33:33.063830 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:33.063784 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bda3dd4caecd91b009836dc846c97c6ebfa769567068d1a60cd431c084e8a90"} err="failed to get container status \"0bda3dd4caecd91b009836dc846c97c6ebfa769567068d1a60cd431c084e8a90\": rpc error: code = NotFound desc = could not find container \"0bda3dd4caecd91b009836dc846c97c6ebfa769567068d1a60cd431c084e8a90\": container with ID starting with 0bda3dd4caecd91b009836dc846c97c6ebfa769567068d1a60cd431c084e8a90 not found: ID does not exist" Jul 16 17:33:33.080774 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:33.080747 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm"] Jul 16 17:33:33.087335 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:33.087314 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b44c0f-744d984fd7-fvzsm"] Jul 16 17:33:34.470755 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:34.470728 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:33:34.909752 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:34.909717 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2a55619-2afc-4257-b199-c732a5dc658b" path="/var/lib/kubelet/pods/b2a55619-2afc-4257-b199-c732a5dc658b/volumes" Jul 16 17:33:35.689676 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:35.689638 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:33:40.030113 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:40.030061 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:33:40.131086 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:40.131053 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:33:40.343227 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:40.343149 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:33:40.576549 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:40.576514 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:33:41.080699 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:41.080665 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:33:48.095363 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:48.095332 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6"] Jul 16 17:33:48.095755 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:48.095531 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" podUID="a2b227b6-56ee-4ae3-927c-0f62ca637f37" containerName="maas-api" containerID="cri-o://6f247d609f679daf87347213b783a1f24865dded808bf6f5c1e06b2eedc66c1b" gracePeriod=30 Jul 16 17:33:48.120487 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:48.120451 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:33:48.347990 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:48.347928 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" Jul 16 17:33:48.426914 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:48.426888 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl55l\" (UniqueName: \"kubernetes.io/projected/a2b227b6-56ee-4ae3-927c-0f62ca637f37-kube-api-access-tl55l\") pod \"a2b227b6-56ee-4ae3-927c-0f62ca637f37\" (UID: \"a2b227b6-56ee-4ae3-927c-0f62ca637f37\") " Jul 16 17:33:48.427053 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:48.426920 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b227b6-56ee-4ae3-927c-0f62ca637f37-maas-api-tls\") pod \"a2b227b6-56ee-4ae3-927c-0f62ca637f37\" (UID: \"a2b227b6-56ee-4ae3-927c-0f62ca637f37\") " Jul 16 17:33:48.427053 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:48.426942 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b227b6-56ee-4ae3-927c-0f62ca637f37-openshift-service-ca\") pod \"a2b227b6-56ee-4ae3-927c-0f62ca637f37\" (UID: \"a2b227b6-56ee-4ae3-927c-0f62ca637f37\") " Jul 16 17:33:48.427387 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:48.427356 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b227b6-56ee-4ae3-927c-0f62ca637f37-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a2b227b6-56ee-4ae3-927c-0f62ca637f37" (UID: "a2b227b6-56ee-4ae3-927c-0f62ca637f37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:33:48.428948 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:48.428925 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2b227b6-56ee-4ae3-927c-0f62ca637f37-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a2b227b6-56ee-4ae3-927c-0f62ca637f37" (UID: "a2b227b6-56ee-4ae3-927c-0f62ca637f37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:33:48.429041 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:48.428997 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b227b6-56ee-4ae3-927c-0f62ca637f37-kube-api-access-tl55l" (OuterVolumeSpecName: "kube-api-access-tl55l") pod "a2b227b6-56ee-4ae3-927c-0f62ca637f37" (UID: "a2b227b6-56ee-4ae3-927c-0f62ca637f37"). InnerVolumeSpecName "kube-api-access-tl55l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:33:48.452363 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:48.452339 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:33:48.528364 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:48.528337 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl55l\" (UniqueName: \"kubernetes.io/projected/a2b227b6-56ee-4ae3-927c-0f62ca637f37-kube-api-access-tl55l\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:33:48.528364 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:48.528363 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b227b6-56ee-4ae3-927c-0f62ca637f37-maas-api-tls\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:33:48.528364 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:48.528372 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b227b6-56ee-4ae3-927c-0f62ca637f37-openshift-service-ca\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:33:49.130176 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:49.130143 2575 generic.go:356] "Generic (PLEG): container finished" podID="a2b227b6-56ee-4ae3-927c-0f62ca637f37" containerID="6f247d609f679daf87347213b783a1f24865dded808bf6f5c1e06b2eedc66c1b" exitCode=0 Jul 16 17:33:49.130607 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:49.130184 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" Jul 16 17:33:49.130607 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:49.130207 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" event={"ID":"a2b227b6-56ee-4ae3-927c-0f62ca637f37","Type":"ContainerDied","Data":"6f247d609f679daf87347213b783a1f24865dded808bf6f5c1e06b2eedc66c1b"} Jul 16 17:33:49.130607 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:49.130257 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6" event={"ID":"a2b227b6-56ee-4ae3-927c-0f62ca637f37","Type":"ContainerDied","Data":"040efc697ba7ee9450634c0d58a55f0818fc00a28ca3fbb8812a7fd443dd71e6"} Jul 16 17:33:49.130607 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:49.130279 2575 scope.go:117] "RemoveContainer" containerID="6f247d609f679daf87347213b783a1f24865dded808bf6f5c1e06b2eedc66c1b" Jul 16 17:33:49.138499 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:49.138482 2575 scope.go:117] "RemoveContainer" containerID="6f247d609f679daf87347213b783a1f24865dded808bf6f5c1e06b2eedc66c1b" Jul 16 17:33:49.138764 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:33:49.138746 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f247d609f679daf87347213b783a1f24865dded808bf6f5c1e06b2eedc66c1b\": container with ID starting with 6f247d609f679daf87347213b783a1f24865dded808bf6f5c1e06b2eedc66c1b not found: ID does not exist" containerID="6f247d609f679daf87347213b783a1f24865dded808bf6f5c1e06b2eedc66c1b" Jul 16 17:33:49.138809 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:49.138773 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f247d609f679daf87347213b783a1f24865dded808bf6f5c1e06b2eedc66c1b"} err="failed to get container status \"6f247d609f679daf87347213b783a1f24865dded808bf6f5c1e06b2eedc66c1b\": rpc error: code = NotFound desc = could not find container \"6f247d609f679daf87347213b783a1f24865dded808bf6f5c1e06b2eedc66c1b\": container with ID starting with 6f247d609f679daf87347213b783a1f24865dded808bf6f5c1e06b2eedc66c1b not found: ID does not exist" Jul 16 17:33:49.152850 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:49.152829 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6"] Jul 16 17:33:49.160524 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:49.160506 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8d3a22-55f6579477-ggpz6"] Jul 16 17:33:50.909671 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:50.909634 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2b227b6-56ee-4ae3-927c-0f62ca637f37" path="/var/lib/kubelet/pods/a2b227b6-56ee-4ae3-927c-0f62ca637f37/volumes" Jul 16 17:33:51.166597 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:51.166519 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lbtrw"] Jul 16 17:33:55.180607 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:55.180570 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn"] Jul 16 17:33:55.181003 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:55.180795 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" podUID="e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" containerName="istio-proxy" containerID="cri-o://e8daf8cda435b44194e3f8e82019f1f9ec4546cf57e80ee005eb429ef3a40732" gracePeriod=30 Jul 16 17:33:56.325054 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:56.325020 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:33:56.543304 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:33:56.543265 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:34:00.437741 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.437714 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:34:00.536138 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.536015 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-credential-socket\") pod \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " Jul 16 17:34:00.536138 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.536072 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-token\") pod \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " Jul 16 17:34:00.536381 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.536150 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-envoy\") pod \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " Jul 16 17:34:00.536381 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.536173 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-data\") pod \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " Jul 16 17:34:00.536381 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.536209 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d66x8\" (UniqueName: \"kubernetes.io/projected/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-kube-api-access-d66x8\") pod \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " Jul 16 17:34:00.536381 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.536267 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-workload-socket\") pod \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " Jul 16 17:34:00.536381 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.536295 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-workload-certs\") pod \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " Jul 16 17:34:00.536381 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.536339 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istiod-ca-cert\") pod \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " Jul 16 17:34:00.536381 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.536343 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" (UID: "e4b29a1a-5df3-4ccd-8214-e84dc2e857fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:34:00.536381 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.536365 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-podinfo\") pod \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\" (UID: \"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa\") " Jul 16 17:34:00.536801 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.536693 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" (UID: "e4b29a1a-5df3-4ccd-8214-e84dc2e857fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:34:00.536801 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.536697 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-credential-socket\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:34:00.536801 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.536767 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" (UID: "e4b29a1a-5df3-4ccd-8214-e84dc2e857fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:34:00.537016 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.536990 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" (UID: "e4b29a1a-5df3-4ccd-8214-e84dc2e857fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:34:00.537416 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.537391 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-data" (OuterVolumeSpecName: "istio-data") pod "e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" (UID: "e4b29a1a-5df3-4ccd-8214-e84dc2e857fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:34:00.538423 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.538400 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" (UID: "e4b29a1a-5df3-4ccd-8214-e84dc2e857fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:34:00.538523 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.538499 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-kube-api-access-d66x8" (OuterVolumeSpecName: "kube-api-access-d66x8") pod "e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" (UID: "e4b29a1a-5df3-4ccd-8214-e84dc2e857fa"). InnerVolumeSpecName "kube-api-access-d66x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:34:00.538589 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.538534 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" (UID: "e4b29a1a-5df3-4ccd-8214-e84dc2e857fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 17:34:00.538589 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.538577 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" (UID: "e4b29a1a-5df3-4ccd-8214-e84dc2e857fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:34:00.637164 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.637122 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d66x8\" (UniqueName: \"kubernetes.io/projected/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-kube-api-access-d66x8\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:34:00.637164 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.637159 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-workload-socket\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:34:00.637164 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.637173 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-workload-certs\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:34:00.637406 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.637185 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istiod-ca-cert\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:34:00.637406 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.637196 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-podinfo\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:34:00.637406 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.637206 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-token\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:34:00.637406 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.637217 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-envoy\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:34:00.637406 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:00.637226 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa-istio-data\") on node \"ip-10-0-136-129.ec2.internal\" DevicePath \"\"" Jul 16 17:34:01.190219 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:01.190122 2575 generic.go:356] "Generic (PLEG): container finished" podID="e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" containerID="e8daf8cda435b44194e3f8e82019f1f9ec4546cf57e80ee005eb429ef3a40732" exitCode=0 Jul 16 17:34:01.190219 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:01.190174 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" Jul 16 17:34:01.190219 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:01.190187 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" event={"ID":"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa","Type":"ContainerDied","Data":"e8daf8cda435b44194e3f8e82019f1f9ec4546cf57e80ee005eb429ef3a40732"} Jul 16 17:34:01.190459 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:01.190259 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn" event={"ID":"e4b29a1a-5df3-4ccd-8214-e84dc2e857fa","Type":"ContainerDied","Data":"e3aeca43d7b073a39931eab06da2b339e05a8797517f741602e5c59ded9b98d7"} Jul 16 17:34:01.190459 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:01.190290 2575 scope.go:117] "RemoveContainer" containerID="e8daf8cda435b44194e3f8e82019f1f9ec4546cf57e80ee005eb429ef3a40732" Jul 16 17:34:01.198964 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:01.198934 2575 scope.go:117] "RemoveContainer" containerID="8b84fb835f33157d970f055d1639b6af93e136d414b9bcff8e35d287fff1eaa6" Jul 16 17:34:01.206898 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:01.206878 2575 scope.go:117] "RemoveContainer" containerID="e8daf8cda435b44194e3f8e82019f1f9ec4546cf57e80ee005eb429ef3a40732" Jul 16 17:34:01.207201 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:34:01.207181 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8daf8cda435b44194e3f8e82019f1f9ec4546cf57e80ee005eb429ef3a40732\": container with ID starting with e8daf8cda435b44194e3f8e82019f1f9ec4546cf57e80ee005eb429ef3a40732 not found: ID does not exist" containerID="e8daf8cda435b44194e3f8e82019f1f9ec4546cf57e80ee005eb429ef3a40732" Jul 16 17:34:01.207293 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:01.207214 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8daf8cda435b44194e3f8e82019f1f9ec4546cf57e80ee005eb429ef3a40732"} err="failed to get container status \"e8daf8cda435b44194e3f8e82019f1f9ec4546cf57e80ee005eb429ef3a40732\": rpc error: code = NotFound desc = could not find container \"e8daf8cda435b44194e3f8e82019f1f9ec4546cf57e80ee005eb429ef3a40732\": container with ID starting with e8daf8cda435b44194e3f8e82019f1f9ec4546cf57e80ee005eb429ef3a40732 not found: ID does not exist" Jul 16 17:34:01.207293 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:01.207240 2575 scope.go:117] "RemoveContainer" containerID="8b84fb835f33157d970f055d1639b6af93e136d414b9bcff8e35d287fff1eaa6" Jul 16 17:34:01.207512 ip-10-0-136-129 kubenswrapper[2575]: E0716 17:34:01.207494 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b84fb835f33157d970f055d1639b6af93e136d414b9bcff8e35d287fff1eaa6\": container with ID starting with 8b84fb835f33157d970f055d1639b6af93e136d414b9bcff8e35d287fff1eaa6 not found: ID does not exist" containerID="8b84fb835f33157d970f055d1639b6af93e136d414b9bcff8e35d287fff1eaa6" Jul 16 17:34:01.207556 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:01.207518 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b84fb835f33157d970f055d1639b6af93e136d414b9bcff8e35d287fff1eaa6"} err="failed to get container status \"8b84fb835f33157d970f055d1639b6af93e136d414b9bcff8e35d287fff1eaa6\": rpc error: code = NotFound desc = could not find container \"8b84fb835f33157d970f055d1639b6af93e136d414b9bcff8e35d287fff1eaa6\": container with ID starting with 8b84fb835f33157d970f055d1639b6af93e136d414b9bcff8e35d287fff1eaa6 not found: ID does not exist" Jul 16 17:34:01.216357 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:01.216329 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn"] Jul 16 17:34:01.221375 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:01.221352 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-8d3a22-openshift-default-7c588d8959-xt8mn"] Jul 16 17:34:02.165527 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:02.165482 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-5mlts_220953c4-02a2-4397-8e1a-53c65ac397ca/manager/0.log" Jul 16 17:34:02.551253 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:02.551168 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c7f465c76-x245s_83a24043-148b-46c0-977b-9011f90dbb66/copy-manifests/0.log" Jul 16 17:34:02.571923 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:02.571885 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c7f465c76-x245s_83a24043-148b-46c0-977b-9011f90dbb66/manager/0.log" Jul 16 17:34:02.909375 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:02.909345 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" path="/var/lib/kubelet/pods/e4b29a1a-5df3-4ccd-8214-e84dc2e857fa/volumes" Jul 16 17:34:03.086845 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:03.086820 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737035-4xtn5_3f2ffe75-5d81-45c2-b59e-e346e733abf1/cleanup/0.log" Jul 16 17:34:03.217054 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:03.216982 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737050-8s8pn_1774ba3b-813c-4238-bc50-51455341a5c1/cleanup/0.log" Jul 16 17:34:04.984586 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:04.984555 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-pq64f_0a789618-3fda-44e6-a1a1-29ad597510cc/manager/0.log" Jul 16 17:34:05.228463 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:05.228434 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-vlxlv_12c5a5d4-0d98-4acf-9809-267521565c51/registry-server/0.log" Jul 16 17:34:05.477188 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:05.477158 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-lbtrw_b89814e0-9771-4f35-8798-36ab391e9603/limitador/0.log" Jul 16 17:34:05.602607 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:05.602565 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-nq7hv_f98619a8-c497-4579-90f9-e2561701d391/manager/0.log" Jul 16 17:34:05.957425 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:05.957399 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4qk2lg_bc0f6a15-ceb2-40d1-a006-62c585469d63/istio-proxy/0.log" Jul 16 17:34:06.092728 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:06.092701 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jnpsj_713e019f-55eb-4f2c-b168-cde5bde99ecf/discovery/0.log" Jul 16 17:34:06.345342 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:06.345265 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d8b857c67-wx2f8_87dffba0-62da-4093-923b-ab29bfacdcd4/kube-auth-proxy/0.log" Jul 16 17:34:06.478385 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:06.478354 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-sckjm_9b6eccbb-8a9b-46b8-864b-001193c5f1fd/istio-proxy/1.log" Jul 16 17:34:06.779170 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:06.779141 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-bblpd_009da2f0-0152-4212-9e02-404799d12456/payload-processing/0.log" Jul 16 17:34:07.562746 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:07.562717 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-4frr7_1603bec9-df29-4d1a-b070-29fafa963ffe/main/0.log" Jul 16 17:34:13.858179 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:13.858149 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:34:14.423551 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:14.423519 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" limit=1024 Jul 16 17:34:16.955501 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:16.955469 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6hg5p_96a6eb58-3665-40df-a18f-4379194c172c/global-pull-secret-syncer/0.log" Jul 16 17:34:17.263134 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:17.263024 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-chkfd_c111c675-66c3-42dc-8f46-74644b5907a3/konnectivity-agent/0.log" Jul 16 17:34:17.634332 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:17.634303 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-129.ec2.internal_c90abb462952f54ad05f349a2dfb0ab3/haproxy/0.log" Jul 16 17:34:22.260252 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:22.260223 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-pq64f_0a789618-3fda-44e6-a1a1-29ad597510cc/manager/0.log" Jul 16 17:34:22.346077 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:22.346045 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-vlxlv_12c5a5d4-0d98-4acf-9809-267521565c51/registry-server/0.log" Jul 16 17:34:22.499090 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:22.499063 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-lbtrw_b89814e0-9771-4f35-8798-36ab391e9603/limitador/0.log" Jul 16 17:34:22.593077 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:22.592986 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-nq7hv_f98619a8-c497-4579-90f9-e2561701d391/manager/0.log" Jul 16 17:34:24.642741 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:24.642714 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-95brm_29dff70d-ffc3-4068-8d6d-c06ea66733ab/node-exporter/0.log" Jul 16 17:34:24.679041 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:24.679012 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-95brm_29dff70d-ffc3-4068-8d6d-c06ea66733ab/kube-rbac-proxy/0.log" Jul 16 17:34:24.708132 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:24.708071 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-95brm_29dff70d-ffc3-4068-8d6d-c06ea66733ab/init-textfile/0.log" Jul 16 17:34:27.069438 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.069410 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-qvvhc_24e57db4-9ef2-4f43-bb8d-36d2774ac52b/networking-console-plugin/0.log" Jul 16 17:34:27.348680 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.348588 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l"] Jul 16 17:34:27.349077 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.349053 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1774ba3b-813c-4238-bc50-51455341a5c1" containerName="cleanup" Jul 16 17:34:27.349225 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.349211 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1774ba3b-813c-4238-bc50-51455341a5c1" containerName="cleanup" Jul 16 17:34:27.349290 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.349246 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b227b6-56ee-4ae3-927c-0f62ca637f37" containerName="maas-api" Jul 16 17:34:27.349290 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.349256 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b227b6-56ee-4ae3-927c-0f62ca637f37" containerName="maas-api" Jul 16 17:34:27.349290 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.349266 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aeae13c-c29e-4728-a7e7-2a258553f0e8" containerName="cleanup" Jul 16 17:34:27.349290 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.349274 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aeae13c-c29e-4728-a7e7-2a258553f0e8" containerName="cleanup" Jul 16 17:34:27.349290 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.349284 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" containerName="istio-proxy" Jul 16 17:34:27.349535 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.349293 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" containerName="istio-proxy" Jul 16 17:34:27.349535 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.349317 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" containerName="istio-proxy" Jul 16 17:34:27.349535 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.349326 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" containerName="istio-proxy" Jul 16 17:34:27.349535 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.349338 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2a55619-2afc-4257-b199-c732a5dc658b" containerName="maas-api" Jul 16 17:34:27.349535 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.349347 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2a55619-2afc-4257-b199-c732a5dc658b" containerName="maas-api" Jul 16 17:34:27.349535 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.349449 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2a55619-2afc-4257-b199-c732a5dc658b" containerName="maas-api" Jul 16 17:34:27.349535 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.349462 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aeae13c-c29e-4728-a7e7-2a258553f0e8" containerName="cleanup" Jul 16 17:34:27.349535 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.349474 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" containerName="istio-proxy" Jul 16 17:34:27.349535 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.349483 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1774ba3b-813c-4238-bc50-51455341a5c1" containerName="cleanup" Jul 16 17:34:27.349535 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.349493 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b29a1a-5df3-4ccd-8214-e84dc2e857fa" containerName="istio-proxy" Jul 16 17:34:27.349535 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.349502 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b227b6-56ee-4ae3-927c-0f62ca637f37" containerName="maas-api" Jul 16 17:34:27.352661 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.352640 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" Jul 16 17:34:27.355481 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.355461 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p8fjr\"/\"default-dockercfg-nbfnt\"" Jul 16 17:34:27.355587 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.355572 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p8fjr\"/\"openshift-service-ca.crt\"" Jul 16 17:34:27.356430 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.356417 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p8fjr\"/\"kube-root-ca.crt\"" Jul 16 17:34:27.371167 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.371137 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l"] Jul 16 17:34:27.439982 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.439942 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/986fe2fa-4df8-4eb4-9832-5f891be739c6-sys\") pod \"perf-node-gather-daemonset-rkm5l\" (UID: \"986fe2fa-4df8-4eb4-9832-5f891be739c6\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" Jul 16 17:34:27.440198 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.439995 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/986fe2fa-4df8-4eb4-9832-5f891be739c6-proc\") pod \"perf-node-gather-daemonset-rkm5l\" (UID: \"986fe2fa-4df8-4eb4-9832-5f891be739c6\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" Jul 16 17:34:27.440198 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.440012 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/986fe2fa-4df8-4eb4-9832-5f891be739c6-lib-modules\") pod \"perf-node-gather-daemonset-rkm5l\" (UID: \"986fe2fa-4df8-4eb4-9832-5f891be739c6\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" Jul 16 17:34:27.440198 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.440129 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq6lj\" (UniqueName: \"kubernetes.io/projected/986fe2fa-4df8-4eb4-9832-5f891be739c6-kube-api-access-hq6lj\") pod \"perf-node-gather-daemonset-rkm5l\" (UID: \"986fe2fa-4df8-4eb4-9832-5f891be739c6\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" Jul 16 17:34:27.440198 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.440169 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/986fe2fa-4df8-4eb4-9832-5f891be739c6-podres\") pod \"perf-node-gather-daemonset-rkm5l\" (UID: \"986fe2fa-4df8-4eb4-9832-5f891be739c6\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" Jul 16 17:34:27.541158 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.541120 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/986fe2fa-4df8-4eb4-9832-5f891be739c6-sys\") pod \"perf-node-gather-daemonset-rkm5l\" (UID: \"986fe2fa-4df8-4eb4-9832-5f891be739c6\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" Jul 16 17:34:27.541360 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.541170 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/986fe2fa-4df8-4eb4-9832-5f891be739c6-proc\") pod \"perf-node-gather-daemonset-rkm5l\" (UID: \"986fe2fa-4df8-4eb4-9832-5f891be739c6\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" Jul 16 17:34:27.541360 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.541186 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/986fe2fa-4df8-4eb4-9832-5f891be739c6-lib-modules\") pod \"perf-node-gather-daemonset-rkm5l\" (UID: \"986fe2fa-4df8-4eb4-9832-5f891be739c6\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" Jul 16 17:34:27.541360 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.541218 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq6lj\" (UniqueName: \"kubernetes.io/projected/986fe2fa-4df8-4eb4-9832-5f891be739c6-kube-api-access-hq6lj\") pod \"perf-node-gather-daemonset-rkm5l\" (UID: \"986fe2fa-4df8-4eb4-9832-5f891be739c6\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" Jul 16 17:34:27.541360 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.541243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/986fe2fa-4df8-4eb4-9832-5f891be739c6-podres\") pod \"perf-node-gather-daemonset-rkm5l\" (UID: \"986fe2fa-4df8-4eb4-9832-5f891be739c6\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" Jul 16 17:34:27.541360 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.541257 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/986fe2fa-4df8-4eb4-9832-5f891be739c6-proc\") pod \"perf-node-gather-daemonset-rkm5l\" (UID: \"986fe2fa-4df8-4eb4-9832-5f891be739c6\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" Jul 16 17:34:27.541360 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.541257 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/986fe2fa-4df8-4eb4-9832-5f891be739c6-sys\") pod \"perf-node-gather-daemonset-rkm5l\" (UID: \"986fe2fa-4df8-4eb4-9832-5f891be739c6\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" Jul 16 17:34:27.541360 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.541353 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/986fe2fa-4df8-4eb4-9832-5f891be739c6-podres\") pod \"perf-node-gather-daemonset-rkm5l\" (UID: \"986fe2fa-4df8-4eb4-9832-5f891be739c6\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" Jul 16 17:34:27.541637 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.541364 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/986fe2fa-4df8-4eb4-9832-5f891be739c6-lib-modules\") pod \"perf-node-gather-daemonset-rkm5l\" (UID: \"986fe2fa-4df8-4eb4-9832-5f891be739c6\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" Jul 16 17:34:27.556684 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.556665 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq6lj\" (UniqueName: \"kubernetes.io/projected/986fe2fa-4df8-4eb4-9832-5f891be739c6-kube-api-access-hq6lj\") pod \"perf-node-gather-daemonset-rkm5l\" (UID: \"986fe2fa-4df8-4eb4-9832-5f891be739c6\") " pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" Jul 16 17:34:27.662525 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.662498 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" Jul 16 17:34:27.798025 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:27.798000 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l"] Jul 16 17:34:27.800252 ip-10-0-136-129 kubenswrapper[2575]: W0716 17:34:27.800226 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod986fe2fa_4df8_4eb4_9832_5f891be739c6.slice/crio-942f5795692d2ae49d1779dbe471ae10b8f575645eda23e9f694ce7545b2beb4 WatchSource:0}: Error finding container 942f5795692d2ae49d1779dbe471ae10b8f575645eda23e9f694ce7545b2beb4: Status 404 returned error can't find the container with id 942f5795692d2ae49d1779dbe471ae10b8f575645eda23e9f694ce7545b2beb4 Jul 16 17:34:28.310034 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:28.309935 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" event={"ID":"986fe2fa-4df8-4eb4-9832-5f891be739c6","Type":"ContainerStarted","Data":"ebfc88b53b6a6345b2e27212442d94ab32434b295eebd7bef534a1cbf42a2734"} Jul 16 17:34:28.310034 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:28.309993 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" event={"ID":"986fe2fa-4df8-4eb4-9832-5f891be739c6","Type":"ContainerStarted","Data":"942f5795692d2ae49d1779dbe471ae10b8f575645eda23e9f694ce7545b2beb4"} Jul 16 17:34:28.310034 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:28.310020 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" Jul 16 17:34:28.334377 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:28.334325 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" podStartSLOduration=1.334308473 podStartE2EDuration="1.334308473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:28.332015681 +0000 UTC m=+2605.850740915" watchObservedRunningTime="2026-07-16 17:34:28.334308473 +0000 UTC m=+2605.853033706" Jul 16 17:34:30.379571 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:30.379542 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bzwp5_34ee6e6b-9608-4c5b-b556-1676477fe609/dns/0.log" Jul 16 17:34:30.406282 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:30.406253 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bzwp5_34ee6e6b-9608-4c5b-b556-1676477fe609/kube-rbac-proxy/0.log" Jul 16 17:34:30.494932 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:30.494905 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bgc7n_11336a91-8f10-40c6-943e-9383f8082593/dns-node-resolver/0.log" Jul 16 17:34:31.193411 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:31.193379 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-644d85d8c9-qphqg_bfb8d84a-c5c9-465f-969b-626b6187f3a9/registry/0.log" Jul 16 17:34:31.298006 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:31.297975 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qvqn5_bba4b37b-cf6a-4dde-afdb-eb2b7f9f9989/node-ca/0.log" Jul 16 17:34:32.339249 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:32.339218 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4qk2lg_bc0f6a15-ceb2-40d1-a006-62c585469d63/istio-proxy/0.log" Jul 16 17:34:32.456748 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:32.456715 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jnpsj_713e019f-55eb-4f2c-b168-cde5bde99ecf/discovery/0.log" Jul 16 17:34:32.533434 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:32.533410 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d8b857c67-wx2f8_87dffba0-62da-4093-923b-ab29bfacdcd4/kube-auth-proxy/0.log" Jul 16 17:34:32.576512 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:32.576484 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-sckjm_9b6eccbb-8a9b-46b8-864b-001193c5f1fd/istio-proxy/0.log" Jul 16 17:34:32.610578 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:32.610546 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-sckjm_9b6eccbb-8a9b-46b8-864b-001193c5f1fd/istio-proxy/1.log" Jul 16 17:34:32.693138 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:32.693091 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-bblpd_009da2f0-0152-4212-9e02-404799d12456/payload-processing/0.log" Jul 16 17:34:33.498154 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:33.498122 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t7p4b_cf501ec7-3530-4abf-bb14-e1f7aff7c53c/serve-healthcheck-canary/0.log" Jul 16 17:34:34.277521 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:34.277490 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xql75_e8c09583-4113-4f64-974f-03c3047db739/kube-rbac-proxy/0.log" Jul 16 17:34:34.309390 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:34.309364 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xql75_e8c09583-4113-4f64-974f-03c3047db739/exporter/0.log" Jul 16 17:34:34.323510 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:34.323484 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p8fjr/perf-node-gather-daemonset-rkm5l" Jul 16 17:34:34.347986 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:34.347955 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xql75_e8c09583-4113-4f64-974f-03c3047db739/extractor/0.log" Jul 16 17:34:37.143927 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:37.143884 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4qk2lg_bc0f6a15-ceb2-40d1-a006-62c585469d63/istio-proxy/0.log" Jul 16 17:34:37.271786 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:37.271754 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jnpsj_713e019f-55eb-4f2c-b168-cde5bde99ecf/discovery/0.log" Jul 16 17:34:37.305861 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:37.305835 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d8b857c67-wx2f8_87dffba0-62da-4093-923b-ab29bfacdcd4/kube-auth-proxy/0.log" Jul 16 17:34:37.331417 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:37.331390 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-sckjm_9b6eccbb-8a9b-46b8-864b-001193c5f1fd/istio-proxy/0.log" Jul 16 17:34:37.359412 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:37.359385 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-sckjm_9b6eccbb-8a9b-46b8-864b-001193c5f1fd/istio-proxy/1.log" Jul 16 17:34:37.441498 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:37.441466 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-bblpd_009da2f0-0152-4212-9e02-404799d12456/payload-processing/0.log" Jul 16 17:34:37.543339 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:37.543259 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-5mlts_220953c4-02a2-4397-8e1a-53c65ac397ca/manager/0.log" Jul 16 17:34:37.736319 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:37.736289 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c7f465c76-x245s_83a24043-148b-46c0-977b-9011f90dbb66/manager/0.log" Jul 16 17:34:37.758026 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:37.758001 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c7f465c76-x245s_83a24043-148b-46c0-977b-9011f90dbb66/copy-manifests/0.log" Jul 16 17:34:38.654067 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:38.654036 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4qk2lg_bc0f6a15-ceb2-40d1-a006-62c585469d63/istio-proxy/0.log" Jul 16 17:34:38.762842 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:38.762761 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jnpsj_713e019f-55eb-4f2c-b168-cde5bde99ecf/discovery/0.log" Jul 16 17:34:38.795981 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:38.795956 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d8b857c67-wx2f8_87dffba0-62da-4093-923b-ab29bfacdcd4/kube-auth-proxy/0.log" Jul 16 17:34:38.820149 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:38.820116 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-sckjm_9b6eccbb-8a9b-46b8-864b-001193c5f1fd/istio-proxy/0.log" Jul 16 17:34:38.847396 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:38.847363 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-sckjm_9b6eccbb-8a9b-46b8-864b-001193c5f1fd/istio-proxy/1.log" Jul 16 17:34:38.904258 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:38.904228 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-bblpd_009da2f0-0152-4212-9e02-404799d12456/payload-processing/0.log" Jul 16 17:34:44.308724 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:44.308695 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-mxnxv_ae61bb97-d7ee-4bc0-803d-82a5ad11d73d/migrator/0.log" Jul 16 17:34:44.336434 ip-10-0-136-129 kubenswrapper[2575]: I0716 17:34:44.336408 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-mxnxv_ae61bb97-d7ee-4bc0-803d-82a5ad11d73d/graceful-termination/0.log"