Jul 17 21:25:32.178799 ip-10-0-130-56 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 17 21:25:32.179858 ip-10-0-130-56 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 17 21:25:32.179874 ip-10-0-130-56 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 17 21:25:32.180208 ip-10-0-130-56 systemd[1]: Failed to start Kubernetes Kubelet. Jul 17 21:25:42.338793 ip-10-0-130-56 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 17 21:25:42.338810 ip-10-0-130-56 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot c015fcc139894f248f4ac4b5c760467f -- Jul 17 21:27:37.029391 ip-10-0-130-56 systemd[1]: Starting Kubernetes Kubelet... Jul 17 21:27:37.361954 ip-10-0-130-56 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 21:27:37.361954 ip-10-0-130-56 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 17 21:27:37.361954 ip-10-0-130-56 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 21:27:37.361954 ip-10-0-130-56 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 17 21:27:37.361954 ip-10-0-130-56 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 21:27:37.362746 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.362656 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 17 21:27:37.371230 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371203 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:27:37.371230 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371226 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:27:37.371230 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371230 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:27:37.371230 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371233 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:27:37.371230 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371236 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:27:37.371230 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371239 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371243 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371246 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371249 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371252 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371254 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371257 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371262 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371266 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371270 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371272 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371275 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371279 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371283 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371285 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371288 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371290 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371293 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371295 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371298 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:27:37.371462 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371301 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371303 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371306 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371308 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371311 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371313 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371319 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371321 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371324 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371327 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371329 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371332 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371334 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371336 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371340 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371342 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371345 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371347 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371350 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371352 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:27:37.371944 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371354 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:27:37.372491 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371357 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:27:37.372491 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371359 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:27:37.372491 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371362 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:27:37.372491 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371365 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:27:37.372491 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371373 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:27:37.372491 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371377 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:27:37.372491 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371380 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:27:37.372491 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371382 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:27:37.372491 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371385 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:27:37.372491 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371387 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:27:37.372491 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371394 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:27:37.372491 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371397 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:27:37.372491 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371399 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:27:37.372491 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371401 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:27:37.372491 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371404 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:27:37.372491 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371406 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:27:37.372491 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371409 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:27:37.372491 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371411 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:27:37.372491 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371413 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:27:37.372929 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371416 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:27:37.372929 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371418 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:27:37.372929 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371421 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:27:37.372929 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371423 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:27:37.372929 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371426 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:27:37.372929 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371428 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:27:37.372929 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371434 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:27:37.372929 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371438 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:27:37.372929 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371441 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:27:37.372929 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371444 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:27:37.372929 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371448 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:27:37.372929 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371451 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:27:37.372929 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371453 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:27:37.372929 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371456 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:27:37.372929 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371458 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:27:37.372929 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371460 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:27:37.372929 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371463 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:27:37.372929 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371465 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:27:37.372929 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371469 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:27:37.373404 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371472 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:27:37.373404 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371475 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:27:37.373404 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371478 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:27:37.373404 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371480 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:27:37.373404 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371483 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:27:37.373404 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371486 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:27:37.373404 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371488 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:27:37.373404 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371491 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:27:37.373404 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371494 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:27:37.373404 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371497 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:27:37.373404 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371499 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:27:37.373404 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371502 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:27:37.373404 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371504 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:27:37.373404 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371506 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:27:37.373404 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.371509 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:27:37.373404 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372258 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:27:37.373404 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372266 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:27:37.373404 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372269 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:27:37.373404 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372272 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372275 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372278 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372280 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372283 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372286 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372288 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372291 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372293 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372296 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372299 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372301 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372303 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372306 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372309 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372311 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372314 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372316 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372319 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372322 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:27:37.373857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372324 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:27:37.374380 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372327 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:27:37.374380 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372329 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:27:37.374380 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372332 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:27:37.374380 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372334 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:27:37.374380 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372337 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:27:37.374380 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372339 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:27:37.374380 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372342 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:27:37.374380 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372345 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:27:37.374380 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372348 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:27:37.374380 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372352 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:27:37.374380 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372355 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:27:37.374380 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372357 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:27:37.374380 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372360 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:27:37.374380 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372363 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:27:37.374380 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372367 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:27:37.374380 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372369 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:27:37.374380 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372372 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:27:37.374380 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372374 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:27:37.374380 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372377 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:27:37.374876 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372379 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:27:37.374876 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372382 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:27:37.374876 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372384 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:27:37.374876 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372387 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:27:37.374876 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372389 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:27:37.374876 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372392 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:27:37.374876 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372394 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:27:37.374876 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372397 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:27:37.374876 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372400 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:27:37.374876 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372402 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:27:37.374876 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372406 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:27:37.374876 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372408 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:27:37.374876 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372411 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:27:37.374876 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372413 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:27:37.374876 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372416 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:27:37.374876 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372418 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:27:37.374876 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372421 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:27:37.374876 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372423 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:27:37.374876 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372425 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372428 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372430 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372433 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372436 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372438 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372441 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372443 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372446 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372449 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372451 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372454 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372456 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372458 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372461 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372463 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372466 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372468 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372471 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372475 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:27:37.375356 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372479 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:27:37.375857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372482 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:27:37.375857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372486 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:27:37.375857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372489 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:27:37.375857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372492 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:27:37.375857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372496 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:27:37.375857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372499 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:27:37.375857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372502 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:27:37.375857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372505 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:27:37.375857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372508 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:27:37.375857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372511 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:27:37.375857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372514 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:27:37.375857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372517 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:27:37.375857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372520 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:27:37.375857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372522 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:27:37.375857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372525 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:27:37.375857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372528 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:27:37.375857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372530 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:27:37.375857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372920 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:27:37.375857 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372929 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372933 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372936 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372939 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372942 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372945 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372948 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372951 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372953 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372957 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372961 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372964 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372967 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372970 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372972 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372981 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372983 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372986 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372989 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372992 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:27:37.376338 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372994 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.372996 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373000 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373003 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373005 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373008 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373010 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373014 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373016 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373019 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373021 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373023 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373026 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373029 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373032 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373034 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373036 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373039 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373041 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373043 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:27:37.376826 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373046 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:27:37.377337 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373048 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:27:37.377337 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373050 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:27:37.377337 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373053 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:27:37.377337 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373055 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:27:37.377337 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373057 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:27:37.377337 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373060 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:27:37.377337 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373062 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:27:37.377337 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373065 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:27:37.377337 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373069 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:27:37.377337 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373088 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:27:37.377337 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373091 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:27:37.377337 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373094 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:27:37.377337 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373097 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:27:37.377337 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373100 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:27:37.377337 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373103 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:27:37.377337 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373106 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:27:37.377337 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373108 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:27:37.377337 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373111 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:27:37.377337 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373114 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373117 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373120 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373122 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373125 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373128 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373131 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373134 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373136 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373140 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373143 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373146 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373148 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373150 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373153 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373156 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373158 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373161 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373163 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373166 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:27:37.377805 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373169 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373171 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373174 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373177 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373180 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373182 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373184 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373187 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373189 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373192 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373194 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373196 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373199 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373201 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373204 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373206 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373208 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373211 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373214 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373286 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373294 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 17 21:27:37.378304 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373300 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373305 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373309 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373313 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373317 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373322 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373325 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373328 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373331 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373335 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373338 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373341 2580 flags.go:64] FLAG: --cgroup-root="" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373346 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373349 2580 flags.go:64] FLAG: --client-ca-file="" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373354 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373358 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373362 2580 flags.go:64] FLAG: --cluster-domain="" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373365 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373370 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373373 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373379 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373383 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373386 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 17 21:27:37.378809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373389 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373407 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373411 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373415 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373418 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373421 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373424 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373429 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373432 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373436 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373438 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373441 2580 flags.go:64] FLAG: --enable-server="true" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373444 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373448 2580 flags.go:64] FLAG: --event-burst="100" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373451 2580 flags.go:64] FLAG: --event-qps="50" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373454 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373457 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373460 2580 flags.go:64] FLAG: --eviction-hard="" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373464 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373466 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373469 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373474 2580 flags.go:64] FLAG: --eviction-soft="" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373477 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373480 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373483 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 17 21:27:37.379376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373486 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373489 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373492 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373494 2580 flags.go:64] FLAG: --feature-gates="" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373498 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373501 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373504 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373507 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373510 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373513 2580 flags.go:64] FLAG: --help="false" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373515 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-130-56.ec2.internal" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373518 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373523 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373525 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373529 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373532 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373535 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373538 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373540 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373543 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373546 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373549 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373552 2580 flags.go:64] FLAG: --kube-reserved="" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373555 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 17 21:27:37.380095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373557 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373560 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373563 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373566 2580 flags.go:64] FLAG: --lock-file="" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373570 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373573 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373576 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373581 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373585 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373587 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373590 2580 flags.go:64] FLAG: --logging-format="text" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373593 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373596 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373599 2580 flags.go:64] FLAG: --manifest-url="" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373602 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373606 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373608 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373612 2580 flags.go:64] FLAG: --max-pods="110" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373615 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373618 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373623 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373626 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373629 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373632 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373635 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 17 21:27:37.380700 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373643 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373645 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373649 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373652 2580 flags.go:64] FLAG: --pod-cidr="" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373655 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373660 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373662 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373666 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373669 2580 flags.go:64] FLAG: --port="10250" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373672 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373675 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a5303fcde2ef651e" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373679 2580 flags.go:64] FLAG: --qos-reserved="" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373682 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373685 2580 flags.go:64] FLAG: --register-node="true" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373688 2580 flags.go:64] FLAG: --register-with-taints="" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373692 2580 flags.go:64] FLAG: --registry-burst="10" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373695 2580 flags.go:64] FLAG: --registry-qps="5" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373698 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373701 2580 flags.go:64] FLAG: --reserved-memory="" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373705 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373708 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373710 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373713 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373716 2580 flags.go:64] FLAG: --runonce="false" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373719 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 17 21:27:37.381330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373722 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373725 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373728 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373733 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373735 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373738 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373741 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373744 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373747 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373749 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373752 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373755 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373758 2580 flags.go:64] FLAG: --system-cgroups="" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373760 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373765 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373768 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373771 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373779 2580 flags.go:64] FLAG: --tls-min-version="" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373783 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373786 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373789 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373792 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373795 2580 flags.go:64] FLAG: --v="2" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373798 2580 flags.go:64] FLAG: --version="false" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373803 2580 flags.go:64] FLAG: --vmodule="" Jul 17 21:27:37.381940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373807 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.373810 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373925 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373929 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373933 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373935 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373938 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373941 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373943 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373946 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373950 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373952 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373955 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373958 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373960 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373963 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373965 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373968 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373970 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373972 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:27:37.382564 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373975 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373978 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373980 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373983 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373985 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373989 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373991 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373994 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373996 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.373999 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374001 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374004 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374007 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374010 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374012 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374014 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374017 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374019 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374022 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374024 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:27:37.383043 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374027 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374029 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374033 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374035 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374038 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374040 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374043 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374045 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374047 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374050 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374053 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374055 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374058 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374060 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374062 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374065 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374067 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374070 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374086 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374090 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:27:37.383543 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374092 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:27:37.384087 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374095 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:27:37.384087 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374098 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:27:37.384087 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374100 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:27:37.384087 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374102 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:27:37.384087 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374105 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:27:37.384087 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374108 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:27:37.384087 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374112 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:27:37.384087 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374116 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:27:37.384087 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374119 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:27:37.384087 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374122 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:27:37.384087 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374124 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:27:37.384087 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374127 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:27:37.384087 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374129 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:27:37.384087 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374134 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:27:37.384087 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374137 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:27:37.384087 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374140 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:27:37.384087 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374143 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:27:37.384087 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374146 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374149 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374152 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374155 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374157 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374160 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374162 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374165 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374167 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374170 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374172 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374175 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374177 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374180 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374183 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374185 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374188 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374191 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374193 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374197 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:27:37.384525 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374205 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:27:37.385007 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374208 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:27:37.385007 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.374211 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:27:37.385007 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.374698 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 21:27:37.385007 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.382103 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 17 21:27:37.385007 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.382117 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 17 21:27:37.385007 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.382808 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 17 21:27:37.385007 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.382815 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 17 21:27:37.385007 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382886 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:27:37.385007 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382892 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:27:37.385007 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382895 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:27:37.385007 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382898 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:27:37.385007 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382901 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:27:37.385007 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382904 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:27:37.385007 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382907 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382909 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382912 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382914 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382917 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382920 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382922 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382925 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382927 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382929 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382932 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382934 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382937 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382939 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382942 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382944 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382947 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382949 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382952 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382955 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382957 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:27:37.385489 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382960 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:27:37.385985 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382963 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:27:37.385985 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382967 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:27:37.385985 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382970 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:27:37.385985 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382972 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:27:37.385985 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382975 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:27:37.385985 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382977 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:27:37.385985 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382980 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:27:37.385985 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382984 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:27:37.385985 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382987 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:27:37.385985 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382991 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:27:37.385985 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382993 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:27:37.385985 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382996 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:27:37.385985 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.382999 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:27:37.385985 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383001 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:27:37.385985 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383004 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:27:37.385985 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383006 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:27:37.385985 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383009 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:27:37.385985 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383012 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:27:37.385985 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383015 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383017 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383019 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383022 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383024 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383027 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383029 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383033 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383035 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383038 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383040 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383043 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383045 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383048 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383052 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383055 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383059 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383062 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383064 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383067 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:27:37.386457 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383070 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383088 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383094 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383097 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383099 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383102 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383104 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383107 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383110 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383113 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383115 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383118 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383120 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383123 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383126 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383128 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383131 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383133 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383136 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383139 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:27:37.386939 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383142 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:27:37.387463 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383144 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:27:37.387463 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383147 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:27:37.387463 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383149 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:27:37.387463 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383152 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:27:37.387463 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383155 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:27:37.387463 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383157 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:27:37.387463 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383160 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:27:37.387463 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383162 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:27:37.387463 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383166 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:27:37.387463 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383170 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:27:37.387463 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383173 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:27:37.387463 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383176 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:27:37.387463 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.383181 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 21:27:37.387463 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383321 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383327 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383330 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383333 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383337 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383340 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383343 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383346 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383348 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383350 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383354 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383357 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383359 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383361 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383364 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383367 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383369 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383372 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383375 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383378 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:27:37.387817 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383380 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383383 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383385 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383388 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383390 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383393 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383395 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383398 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383400 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383403 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383405 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383408 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383410 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383413 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383415 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383418 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383420 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383422 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383425 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383427 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:27:37.388311 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383429 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383432 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383434 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383436 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383439 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383441 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383443 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383446 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383448 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383451 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383453 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383456 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383458 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383462 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383465 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383468 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383471 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383474 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383477 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383479 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:27:37.388790 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383482 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383485 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383487 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383490 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383492 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383495 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383499 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383502 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383504 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383507 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383509 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383512 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383514 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383516 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383519 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383521 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383524 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383526 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383529 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383531 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:27:37.389371 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383534 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:27:37.389843 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383538 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:27:37.389843 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383541 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:27:37.389843 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383545 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:27:37.389843 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383548 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:27:37.389843 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383550 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:27:37.389843 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383553 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:27:37.389843 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383555 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:27:37.389843 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383558 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:27:37.389843 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383560 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:27:37.389843 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383563 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:27:37.389843 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383565 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:27:37.389843 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383568 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:27:37.389843 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383570 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:27:37.389843 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383573 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:27:37.389843 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383575 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:27:37.389843 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383578 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:27:37.389843 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383580 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:27:37.389843 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:37.383583 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:27:37.390304 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.383588 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 21:27:37.390304 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.383719 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 17 21:27:37.390304 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.386104 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 17 21:27:37.390304 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.387283 2580 server.go:1019] "Starting client certificate rotation" Jul 17 21:27:37.390304 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.387429 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 21:27:37.390304 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.387464 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 21:27:37.390304 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.389237 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 21:27:37.392933 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.392912 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 21:27:37.401295 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.401278 2580 log.go:25] "Validated CRI v1 runtime API" Jul 17 21:27:37.405009 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.404990 2580 log.go:25] "Validated CRI v1 image API" Jul 17 21:27:37.406222 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.406193 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 17 21:27:37.408384 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.408358 2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8bce8a4d-530a-458f-9feb-05e2bca73ea4:/dev/nvme0n1p4 ff8cf89f-7c7b-4927-ab5c-3748b2798678:/dev/nvme0n1p3] Jul 17 21:27:37.408429 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.408385 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 17 21:27:37.412624 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.412489 2580 manager.go:216] Machine: {Timestamp:2026-07-17 21:27:37.411964121 +0000 UTC m=+0.279996704 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102658 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24a4c4117e276cda2f806d2b79bed6 SystemUUID:ec24a4c4-117e-276c-da2f-806d2b79bed6 BootID:c015fcc1-3989-4f24-8f4a-c4b5c760467f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bd:ef:a3:31:49 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bd:ef:a3:31:49 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:a0:92:41:29:82 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 17 21:27:37.412624 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.412618 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 17 21:27:37.412773 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.412759 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 17 21:27:37.414955 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.414927 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 17 21:27:37.415099 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.414959 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-56.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 17 21:27:37.415619 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.415607 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 17 21:27:37.415664 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.415622 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 17 21:27:37.415664 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.415620 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 21:27:37.415746 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.415632 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 21:27:37.415746 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.415704 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 21:27:37.416680 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.416667 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 17 21:27:37.420154 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.420140 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 17 21:27:37.420294 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.420282 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 17 21:27:37.420364 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.420356 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 17 21:27:37.420415 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.420377 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 17 21:27:37.420415 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.420393 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 17 21:27:37.420415 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.420408 2580 kubelet.go:398] "Adding apiserver pod source" Jul 17 21:27:37.420532 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.420420 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 17 21:27:37.420847 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.420834 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 21:27:37.420927 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.420861 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 21:27:37.422989 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.422973 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 17 21:27:37.424471 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.424457 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 17 21:27:37.424520 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.424489 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 17 21:27:37.425458 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.425446 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 17 21:27:37.425528 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.425463 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 17 21:27:37.425528 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.425469 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 17 21:27:37.425528 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.425474 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 17 21:27:37.425528 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.425483 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 17 21:27:37.425528 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.425488 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 17 21:27:37.425528 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.425496 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 17 21:27:37.425528 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.425501 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 17 21:27:37.425528 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.425507 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 17 21:27:37.425528 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.425513 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 17 21:27:37.425528 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.425521 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 17 21:27:37.425528 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.425526 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 17 21:27:37.426261 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.426250 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 17 21:27:37.426303 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.426263 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 17 21:27:37.429584 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.429569 2580 server.go:1268] "Started kubelet" Jul 17 21:27:37.429694 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.429648 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 17 21:27:37.429828 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.429774 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 17 21:27:37.429938 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.429852 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 17 21:27:37.430525 ip-10-0-130-56 systemd[1]: Started Kubernetes Kubelet. Jul 17 21:27:37.431363 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.431342 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-56.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 17 21:27:37.431462 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.431416 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 17 21:27:37.431977 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:37.431948 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-56.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 17 21:27:37.431977 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:37.431947 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 17 21:27:37.432100 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.431997 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 17 21:27:37.434428 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:37.433766 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-56.ec2.internal.18c330e9a569a161 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-56.ec2.internal,UID:ip-10-0-130-56.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-56.ec2.internal,},FirstTimestamp:2026-07-17 21:27:37.429524833 +0000 UTC m=+0.297557396,LastTimestamp:2026-07-17 21:27:37.429524833 +0000 UTC m=+0.297557396,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-56.ec2.internal,}" Jul 17 21:27:37.435068 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:37.435052 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 17 21:27:37.435836 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.435821 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 17 21:27:37.435898 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.435837 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 17 21:27:37.436320 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:37.435962 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-56.ec2.internal\" not found" Jul 17 21:27:37.436516 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.436382 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 17 21:27:37.436516 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.436423 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 17 21:27:37.436516 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.436439 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 17 21:27:37.436991 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.436960 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 17 21:27:37.436991 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.436991 2580 factory.go:55] Registering systemd factory Jul 17 21:27:37.437137 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.437002 2580 factory.go:223] Registration of the systemd container factory successfully Jul 17 21:27:37.437447 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.437431 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 17 21:27:37.437501 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.437448 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 17 21:27:37.437596 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.437580 2580 factory.go:153] Registering CRI-O factory Jul 17 21:27:37.437639 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.437627 2580 factory.go:223] Registration of the crio container factory successfully Jul 17 21:27:37.437685 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.437651 2580 factory.go:103] Registering Raw factory Jul 17 21:27:37.437685 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.437664 2580 manager.go:1196] Started watching for new ooms in manager Jul 17 21:27:37.438125 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:37.438060 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 17 21:27:37.438858 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.438841 2580 manager.go:349] Starting recovery of all containers Jul 17 21:27:37.439843 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:37.439797 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-56.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 17 21:27:37.441815 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.441790 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4wmpl" Jul 17 21:27:37.446555 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.446534 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4wmpl" Jul 17 21:27:37.450051 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.450034 2580 manager.go:354] Recovery completed Jul 17 21:27:37.455018 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.455007 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:27:37.456930 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.456915 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:27:37.456989 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.456945 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:27:37.456989 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.456957 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:27:37.457643 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.457623 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 17 21:27:37.457643 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.457642 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 17 21:27:37.457748 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.457659 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 17 21:27:37.457748 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.457631 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:27:37.458441 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.458427 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:27:37.458524 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.458457 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:27:37.458524 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.458470 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:27:37.459598 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.459585 2580 policy_none.go:49] "None policy: Start" Jul 17 21:27:37.459647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.459602 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 17 21:27:37.459647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.459612 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 17 21:27:37.461599 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.461587 2580 policy_none.go:47] "Start" Jul 17 21:27:37.495050 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.495033 2580 manager.go:337] "Starting Device Plugin manager" Jul 17 21:27:37.495159 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:37.495070 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 17 21:27:37.495159 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.495097 2580 server.go:87] "Starting device plugin registration server" Jul 17 21:27:37.495352 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.495341 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 17 21:27:37.495386 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.495354 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 17 21:27:37.495480 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.495452 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 17 21:27:37.495616 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.495532 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 17 21:27:37.495616 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.495541 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 17 21:27:37.496091 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:37.496058 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 17 21:27:37.496186 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:37.496118 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-56.ec2.internal\" not found" Jul 17 21:27:37.569678 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.569639 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 17 21:27:37.571601 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.570953 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 17 21:27:37.571601 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.570988 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 17 21:27:37.571601 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.571013 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 17 21:27:37.571601 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:37.571047 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 17 21:27:37.573404 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.573385 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:27:37.596195 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.596175 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:27:37.597044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.597028 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:27:37.597140 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.597059 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:27:37.597140 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.597071 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:27:37.597140 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.597116 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-56.ec2.internal" Jul 17 21:27:37.605117 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.605100 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-56.ec2.internal" Jul 17 21:27:37.605164 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:37.605126 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-56.ec2.internal\": node \"ip-10-0-130-56.ec2.internal\" not found" Jul 17 21:27:37.618786 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:37.618735 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-56.ec2.internal\" not found" Jul 17 21:27:37.671519 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.671494 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-56.ec2.internal"] Jul 17 21:27:37.671624 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.671561 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:27:37.672911 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.672896 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:27:37.672969 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.672931 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:27:37.672969 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.672945 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:27:37.675228 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.675215 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:27:37.675359 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.675345 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal" Jul 17 21:27:37.675407 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.675375 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:27:37.675980 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.675963 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:27:37.676044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.675997 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:27:37.676044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.676017 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:27:37.676044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.676027 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:27:37.676044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.676031 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:27:37.676192 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.676048 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:27:37.678590 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.678571 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-56.ec2.internal" Jul 17 21:27:37.678681 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.678598 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:27:37.679329 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.679311 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:27:37.679409 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.679347 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:27:37.679409 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.679357 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:27:37.695731 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:37.695709 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-56.ec2.internal\" not found" node="ip-10-0-130-56.ec2.internal" Jul 17 21:27:37.700226 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:37.700205 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-56.ec2.internal\" not found" node="ip-10-0-130-56.ec2.internal" Jul 17 21:27:37.719157 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:37.719139 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-56.ec2.internal\" not found" Jul 17 21:27:37.739681 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.739644 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d1a835e2ecd8bc0f785c433c4eda9ce2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal\" (UID: \"d1a835e2ecd8bc0f785c433c4eda9ce2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal" Jul 17 21:27:37.739794 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.739686 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1a835e2ecd8bc0f785c433c4eda9ce2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal\" (UID: \"d1a835e2ecd8bc0f785c433c4eda9ce2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal" Jul 17 21:27:37.739794 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.739711 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc49aa9b1d03314cbcca016a92d97157-config\") pod \"kube-apiserver-proxy-ip-10-0-130-56.ec2.internal\" (UID: \"dc49aa9b1d03314cbcca016a92d97157\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-56.ec2.internal" Jul 17 21:27:37.820175 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:37.820140 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-56.ec2.internal\" not found" Jul 17 21:27:37.840578 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.840557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d1a835e2ecd8bc0f785c433c4eda9ce2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal\" (UID: \"d1a835e2ecd8bc0f785c433c4eda9ce2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal" Jul 17 21:27:37.840650 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.840584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1a835e2ecd8bc0f785c433c4eda9ce2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal\" (UID: \"d1a835e2ecd8bc0f785c433c4eda9ce2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal" Jul 17 21:27:37.840650 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.840602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc49aa9b1d03314cbcca016a92d97157-config\") pod \"kube-apiserver-proxy-ip-10-0-130-56.ec2.internal\" (UID: \"dc49aa9b1d03314cbcca016a92d97157\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-56.ec2.internal" Jul 17 21:27:37.840650 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.840627 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d1a835e2ecd8bc0f785c433c4eda9ce2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal\" (UID: \"d1a835e2ecd8bc0f785c433c4eda9ce2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal" Jul 17 21:27:37.840650 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.840639 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc49aa9b1d03314cbcca016a92d97157-config\") pod \"kube-apiserver-proxy-ip-10-0-130-56.ec2.internal\" (UID: \"dc49aa9b1d03314cbcca016a92d97157\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-56.ec2.internal" Jul 17 21:27:37.840794 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.840670 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1a835e2ecd8bc0f785c433c4eda9ce2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal\" (UID: \"d1a835e2ecd8bc0f785c433c4eda9ce2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal" Jul 17 21:27:37.920925 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:37.920847 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-56.ec2.internal\" not found" Jul 17 21:27:37.998535 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:37.998488 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal" Jul 17 21:27:38.003583 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:38.003156 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-56.ec2.internal" Jul 17 21:27:38.021579 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:38.021428 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-56.ec2.internal\" not found" Jul 17 21:27:38.121977 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:38.121936 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-56.ec2.internal\" not found" Jul 17 21:27:38.222589 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:38.222529 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-56.ec2.internal\" not found" Jul 17 21:27:38.270643 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:38.270615 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:27:38.322737 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:38.322700 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-56.ec2.internal\" not found" Jul 17 21:27:38.386460 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:38.386411 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 17 21:27:38.386928 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:38.386586 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 21:27:38.386928 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:38.386635 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 21:27:38.422887 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:38.422849 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-56.ec2.internal\" not found" Jul 17 21:27:39.897455 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:39.897417 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 21:22:38 +0000 UTC" deadline="2028-01-22 00:17:37.735916889 +0000 UTC" Jul 17 21:27:39.897455 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:39.897446 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13274h49m57.838473968s" Jul 17 21:27:39.969625 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:39.969606 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-56.ec2.internal\" not found" Jul 17 21:27:40.070552 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:40.070520 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-56.ec2.internal\" not found" Jul 17 21:27:40.093877 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:40.093839 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a835e2ecd8bc0f785c433c4eda9ce2.slice/crio-a41667f308c8bf24298627aa22f023384bc3651defd5019dbdba58baf1256b7a WatchSource:0}: Error finding container a41667f308c8bf24298627aa22f023384bc3651defd5019dbdba58baf1256b7a: Status 404 returned error can't find the container with id a41667f308c8bf24298627aa22f023384bc3651defd5019dbdba58baf1256b7a Jul 17 21:27:40.094202 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:40.094182 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc49aa9b1d03314cbcca016a92d97157.slice/crio-5ee883f8324a5d743816277f9e1fbdb7d31a7f2729f997603e2e42f6975c2150 WatchSource:0}: Error finding container 5ee883f8324a5d743816277f9e1fbdb7d31a7f2729f997603e2e42f6975c2150: Status 404 returned error can't find the container with id 5ee883f8324a5d743816277f9e1fbdb7d31a7f2729f997603e2e42f6975c2150 Jul 17 21:27:40.098271 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.098247 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 21:27:40.170913 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:40.170880 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-56.ec2.internal\" not found" Jul 17 21:27:40.191469 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.191450 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:27:40.271634 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:40.271555 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-56.ec2.internal\" not found" Jul 17 21:27:40.363083 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.363054 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:27:40.382340 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.382315 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal" Jul 17 21:27:40.393301 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.393281 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 21:27:40.394500 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.394487 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-56.ec2.internal" Jul 17 21:27:40.400692 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.400678 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 21:27:40.867214 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.867178 2580 apiserver.go:52] "Watching apiserver" Jul 17 21:27:40.875031 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.875006 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 17 21:27:40.875516 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.875484 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-56.ec2.internal","openshift-cluster-node-tuning-operator/tuned-cshwz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal","openshift-multus/multus-b5zsn","openshift-network-operator/iptables-alerter-h9dwj","openshift-ovn-kubernetes/ovnkube-node-rhkgr","kube-system/global-pull-secret-syncer-fl9qj","kube-system/konnectivity-agent-bvspz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw","openshift-dns/node-resolver-4fgf5","openshift-image-registry/node-ca-6mqj2","openshift-multus/multus-additional-cni-plugins-qlgx8","openshift-multus/network-metrics-daemon-d2x4m","openshift-network-diagnostics/network-check-target-n7c88"] Jul 17 21:27:40.878105 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.878076 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:40.878220 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:40.878160 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:27:40.882941 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.882920 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 17 21:27:40.883884 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.883623 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:40.883884 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.883667 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:27:40.883884 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:40.883702 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:27:40.883884 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:40.883728 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:27:40.885134 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.885117 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b5zsn" Jul 17 21:27:40.888019 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.887349 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h9dwj" Jul 17 21:27:40.888019 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.887550 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 17 21:27:40.888310 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.888288 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 17 21:27:40.888411 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.888335 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 17 21:27:40.888473 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.888291 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 17 21:27:40.888599 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.888582 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hpncw\"" Jul 17 21:27:40.889756 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.889739 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:40.889843 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.889832 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 17 21:27:40.890476 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.890455 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2vcpd\"" Jul 17 21:27:40.890777 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.890695 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 17 21:27:40.890777 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.890738 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 17 21:27:40.892564 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.892545 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:40.892794 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.892774 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 17 21:27:40.893004 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.892987 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bnf27\"" Jul 17 21:27:40.893179 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.893162 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 17 21:27:40.893269 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.893234 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 17 21:27:40.895558 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.894712 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bvspz" Jul 17 21:27:40.896146 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.896128 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 17 21:27:40.896376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.896361 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 17 21:27:40.896732 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.896717 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 17 21:27:40.897168 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.897149 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 17 21:27:40.897229 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.897177 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 17 21:27:40.897517 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.897498 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5wpfx\"" Jul 17 21:27:40.897692 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.897677 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 17 21:27:40.897833 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.897810 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 17 21:27:40.897892 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.897848 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pl4bj\"" Jul 17 21:27:40.898982 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.898111 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 17 21:27:40.898982 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.898252 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 21:27:40.899147 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.899039 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f44bc251-268b-42f5-b9d3-a742151884f9-kubelet-config\") pod \"global-pull-secret-syncer-fl9qj\" (UID: \"f44bc251-268b-42f5-b9d3-a742151884f9\") " pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:40.899147 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.899074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f44bc251-268b-42f5-b9d3-a742151884f9-dbus\") pod \"global-pull-secret-syncer-fl9qj\" (UID: \"f44bc251-268b-42f5-b9d3-a742151884f9\") " pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:40.903912 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.899109 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret\") pod \"global-pull-secret-syncer-fl9qj\" (UID: \"f44bc251-268b-42f5-b9d3-a742151884f9\") " pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:40.903985 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.903937 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm79k\" (UniqueName: \"kubernetes.io/projected/896ed8fb-96fc-4834-83a6-807a16a340b3-kube-api-access-gm79k\") pod \"network-metrics-daemon-d2x4m\" (UID: \"896ed8fb-96fc-4834-83a6-807a16a340b3\") " pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:40.903985 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.903955 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtgm2\" (UniqueName: \"kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2\") pod \"network-check-target-n7c88\" (UID: \"c85213e4-c44c-418e-ae68-c74526ae6fa5\") " pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:27:40.903985 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.903970 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-host-var-lib-kubelet\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:40.903985 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.903985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/759d3e52-70f2-4924-8b94-f06cf30df241-multus-daemon-config\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:40.904109 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.903999 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-multus-cni-dir\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:40.904109 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.904013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-cnibin\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:40.904109 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.904033 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/759d3e52-70f2-4924-8b94-f06cf30df241-cni-binary-copy\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:40.904109 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.904048 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-host-var-lib-cni-bin\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:40.904109 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.904068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-etc-kubernetes\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:40.904109 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.904093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs\") pod \"network-metrics-daemon-d2x4m\" (UID: \"896ed8fb-96fc-4834-83a6-807a16a340b3\") " pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:40.904288 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.904111 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-system-cni-dir\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:40.904288 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.904125 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-multus-socket-dir-parent\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:40.904288 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.904145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-host-run-k8s-cni-cncf-io\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:40.904288 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.904159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-host-var-lib-cni-multus\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:40.904288 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.904174 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-multus-conf-dir\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:40.904288 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.904187 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-os-release\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:40.904288 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.904205 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-host-run-netns\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:40.904288 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.904227 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-hostroot\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:40.904288 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.904250 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-host-run-multus-certs\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:40.904288 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.904267 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9dx4\" (UniqueName: \"kubernetes.io/projected/759d3e52-70f2-4924-8b94-f06cf30df241-kube-api-access-d9dx4\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:40.905598 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.905306 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:40.908133 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.907698 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4fgf5" Jul 17 21:27:40.908328 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.908296 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f2br9\"" Jul 17 21:27:40.908328 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.908325 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 17 21:27:40.908500 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.908334 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 17 21:27:40.911432 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.910057 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 17 21:27:40.911432 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.910296 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 17 21:27:40.911432 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.910584 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hmqpp\"" Jul 17 21:27:40.911432 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.910825 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6mqj2" Jul 17 21:27:40.912648 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.912632 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:40.913136 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.913119 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 17 21:27:40.913421 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.913294 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-762kt\"" Jul 17 21:27:40.913538 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.913357 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 17 21:27:40.913651 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.913365 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 17 21:27:40.914952 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.914931 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 17 21:27:40.915062 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.914940 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c9rvn\"" Jul 17 21:27:40.915256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.915236 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 17 21:27:40.916151 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.916131 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vhrph" Jul 17 21:27:40.925469 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.925447 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vhrph" Jul 17 21:27:40.983542 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:40.983475 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 17 21:27:41.005087 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-node-log\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.005240 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/759d3e52-70f2-4924-8b94-f06cf30df241-cni-binary-copy\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.005240 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005119 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-host-var-lib-cni-bin\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.005240 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005144 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-etc-kubernetes\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.005240 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005166 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-log-socket\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.005240 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005202 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-host-var-lib-cni-bin\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.005240 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005229 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-etc-kubernetes\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.005654 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005271 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-ovn-node-metrics-cert\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.005654 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3e6f72d2-a524-4587-a7f5-278128245a09-hosts-file\") pod \"node-resolver-4fgf5\" (UID: \"3e6f72d2-a524-4587-a7f5-278128245a09\") " pod="openshift-dns/node-resolver-4fgf5" Jul 17 21:27:41.005654 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005332 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs\") pod \"network-metrics-daemon-d2x4m\" (UID: \"896ed8fb-96fc-4834-83a6-807a16a340b3\") " pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:41.005654 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005359 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-multus-socket-dir-parent\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.005654 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005383 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9dx4\" (UniqueName: \"kubernetes.io/projected/759d3e52-70f2-4924-8b94-f06cf30df241-kube-api-access-d9dx4\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.005654 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005424 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd84q\" (UniqueName: \"kubernetes.io/projected/fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694-kube-api-access-cd84q\") pod \"node-ca-6mqj2\" (UID: \"fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694\") " pod="openshift-image-registry/node-ca-6mqj2" Jul 17 21:27:41.005654 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005450 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-run-openvswitch\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.005654 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:41.005453 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:41.005654 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005471 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-sysconfig\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.005654 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005495 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-systemd\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.005654 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-multus-socket-dir-parent\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.005654 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:41.005585 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs podName:896ed8fb-96fc-4834-83a6-807a16a340b3 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:41.505549928 +0000 UTC m=+2.927269159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs") pod "network-metrics-daemon-d2x4m" (UID: "896ed8fb-96fc-4834-83a6-807a16a340b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:41.005654 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-os-release\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.006194 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005677 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-host-run-netns\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.006194 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005719 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-host-run-netns\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.006194 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005691 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/759d3e52-70f2-4924-8b94-f06cf30df241-cni-binary-copy\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.006194 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005743 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc3959d9-8af3-4c63-a611-9fb0321a7e47-agent-certs\") pod \"konnectivity-agent-bvspz\" (UID: \"cc3959d9-8af3-4c63-a611-9fb0321a7e47\") " pod="kube-system/konnectivity-agent-bvspz" Jul 17 21:27:41.006194 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005742 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-os-release\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.006194 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005776 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-sysctl-d\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.006194 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005816 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gvm6\" (UniqueName: \"kubernetes.io/projected/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-kube-api-access-9gvm6\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.006194 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f44bc251-268b-42f5-b9d3-a742151884f9-dbus\") pod \"global-pull-secret-syncer-fl9qj\" (UID: \"f44bc251-268b-42f5-b9d3-a742151884f9\") " pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:41.006194 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005884 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-host-var-lib-kubelet\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.006194 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005908 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-system-cni-dir\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.006194 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005931 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-kubelet\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.006194 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005979 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-host-var-lib-kubelet\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.006194 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.005998 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f44bc251-268b-42f5-b9d3-a742151884f9-dbus\") pod \"global-pull-secret-syncer-fl9qj\" (UID: \"f44bc251-268b-42f5-b9d3-a742151884f9\") " pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:41.006194 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-var-lib-openvswitch\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.006194 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006038 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-ovnkube-config\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.006194 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006056 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-env-overrides\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.006194 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006072 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-sys\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.007013 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006088 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-kubernetes\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.007013 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006128 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-tuned\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.007013 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-system-cni-dir\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.007013 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006209 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-host-var-lib-cni-multus\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.007013 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006233 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chbpv\" (UniqueName: \"kubernetes.io/projected/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-kube-api-access-chbpv\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.007013 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006251 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-system-cni-dir\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.007013 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006254 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-host-var-lib-cni-multus\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.007013 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006255 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-slash\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.007013 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-ovnkube-script-lib\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.007013 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006338 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-var-lib-kubelet\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.007013 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006363 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm79k\" (UniqueName: \"kubernetes.io/projected/896ed8fb-96fc-4834-83a6-807a16a340b3-kube-api-access-gm79k\") pod \"network-metrics-daemon-d2x4m\" (UID: \"896ed8fb-96fc-4834-83a6-807a16a340b3\") " pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:41.007013 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006385 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtgm2\" (UniqueName: \"kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2\") pod \"network-check-target-n7c88\" (UID: \"c85213e4-c44c-418e-ae68-c74526ae6fa5\") " pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:27:41.007013 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006424 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694-host\") pod \"node-ca-6mqj2\" (UID: \"fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694\") " pod="openshift-image-registry/node-ca-6mqj2" Jul 17 21:27:41.007013 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006460 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.007013 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006482 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.007013 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006504 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-run-ovn-kubernetes\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.007678 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-cni-bin\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.007678 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006610 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-run-ovn\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.007678 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006647 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-multus-cni-dir\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.007678 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006668 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-cnibin\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.007678 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006686 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-systemd-units\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.007678 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-run-systemd\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.007678 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006712 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-multus-cni-dir\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.007678 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006725 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-cnibin\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.007678 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfsjf\" (UniqueName: \"kubernetes.io/projected/3e6f72d2-a524-4587-a7f5-278128245a09-kube-api-access-kfsjf\") pod \"node-resolver-4fgf5\" (UID: \"3e6f72d2-a524-4587-a7f5-278128245a09\") " pod="openshift-dns/node-resolver-4fgf5" Jul 17 21:27:41.007678 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006791 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-cni-netd\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.007678 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-host-run-k8s-cni-cncf-io\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.007678 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-multus-conf-dir\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.007678 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006909 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-etc-openvswitch\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.007678 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006909 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-host-run-k8s-cni-cncf-io\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.007678 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006935 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-registration-dir\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.007678 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc3959d9-8af3-4c63-a611-9fb0321a7e47-konnectivity-ca\") pod \"konnectivity-agent-bvspz\" (UID: \"cc3959d9-8af3-4c63-a611-9fb0321a7e47\") " pod="kube-system/konnectivity-agent-bvspz" Jul 17 21:27:41.007678 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006939 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-multus-conf-dir\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.008256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.006991 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6f72d2-a524-4587-a7f5-278128245a09-tmp-dir\") pod \"node-resolver-4fgf5\" (UID: \"3e6f72d2-a524-4587-a7f5-278128245a09\") " pod="openshift-dns/node-resolver-4fgf5" Jul 17 21:27:41.008256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007040 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3ea973f-03c8-4cf1-b45b-b557aef07ef4-host-slash\") pod \"iptables-alerter-h9dwj\" (UID: \"f3ea973f-03c8-4cf1-b45b-b557aef07ef4\") " pod="openshift-network-operator/iptables-alerter-h9dwj" Jul 17 21:27:41.008256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-os-release\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.008256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007102 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-device-dir\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.008256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007130 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret\") pod \"global-pull-secret-syncer-fl9qj\" (UID: \"f44bc251-268b-42f5-b9d3-a742151884f9\") " pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:41.008256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007167 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svr7z\" (UniqueName: \"kubernetes.io/projected/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-kube-api-access-svr7z\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.008256 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:41.007206 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:41.008256 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:41.007245 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret podName:f44bc251-268b-42f5-b9d3-a742151884f9 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:41.507233047 +0000 UTC m=+2.928952258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret") pod "global-pull-secret-syncer-fl9qj" (UID: "f44bc251-268b-42f5-b9d3-a742151884f9") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:41.008256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.008256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007309 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-sys-fs\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.008256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007333 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.008256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007368 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.008256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007431 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-modprobe-d\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.008256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007459 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-sysctl-conf\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.008256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007482 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-tmp\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.008256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-cni-binary-copy\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.008883 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007530 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-socket-dir\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.008883 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007551 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-run\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.008883 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007587 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-hostroot\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.008883 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007612 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-host-run-multus-certs\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.008883 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007631 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694-serviceca\") pod \"node-ca-6mqj2\" (UID: \"fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694\") " pod="openshift-image-registry/node-ca-6mqj2" Jul 17 21:27:41.008883 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007650 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-cnibin\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.008883 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007664 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-run-netns\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.008883 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.008883 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007691 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f44bc251-268b-42f5-b9d3-a742151884f9-kubelet-config\") pod \"global-pull-secret-syncer-fl9qj\" (UID: \"f44bc251-268b-42f5-b9d3-a742151884f9\") " pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:41.008883 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/759d3e52-70f2-4924-8b94-f06cf30df241-multus-daemon-config\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.008883 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007720 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f3ea973f-03c8-4cf1-b45b-b557aef07ef4-iptables-alerter-script\") pod \"iptables-alerter-h9dwj\" (UID: \"f3ea973f-03c8-4cf1-b45b-b557aef07ef4\") " pod="openshift-network-operator/iptables-alerter-h9dwj" Jul 17 21:27:41.008883 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007734 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg2t2\" (UniqueName: \"kubernetes.io/projected/f3ea973f-03c8-4cf1-b45b-b557aef07ef4-kube-api-access-zg2t2\") pod \"iptables-alerter-h9dwj\" (UID: \"f3ea973f-03c8-4cf1-b45b-b557aef07ef4\") " pod="openshift-network-operator/iptables-alerter-h9dwj" Jul 17 21:27:41.008883 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007748 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdgbk\" (UniqueName: \"kubernetes.io/projected/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-kube-api-access-hdgbk\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.008883 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007761 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-lib-modules\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.008883 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007774 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-host\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.008883 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007846 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-hostroot\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.008883 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007867 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/759d3e52-70f2-4924-8b94-f06cf30df241-host-run-multus-certs\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.009718 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.007920 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f44bc251-268b-42f5-b9d3-a742151884f9-kubelet-config\") pod \"global-pull-secret-syncer-fl9qj\" (UID: \"f44bc251-268b-42f5-b9d3-a742151884f9\") " pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:41.009718 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.008222 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/759d3e52-70f2-4924-8b94-f06cf30df241-multus-daemon-config\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.015712 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:41.015686 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:27:41.015821 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:41.015720 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:27:41.015821 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:41.015735 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mtgm2 for pod openshift-network-diagnostics/network-check-target-n7c88: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:41.015821 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:41.015799 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2 podName:c85213e4-c44c-418e-ae68-c74526ae6fa5 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:41.515781252 +0000 UTC m=+2.937500469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mtgm2" (UniqueName: "kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2") pod "network-check-target-n7c88" (UID: "c85213e4-c44c-418e-ae68-c74526ae6fa5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:41.015990 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.015945 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 17 21:27:41.022441 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.022326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm79k\" (UniqueName: \"kubernetes.io/projected/896ed8fb-96fc-4834-83a6-807a16a340b3-kube-api-access-gm79k\") pod \"network-metrics-daemon-d2x4m\" (UID: \"896ed8fb-96fc-4834-83a6-807a16a340b3\") " pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:41.024337 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.023767 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9dx4\" (UniqueName: \"kubernetes.io/projected/759d3e52-70f2-4924-8b94-f06cf30df241-kube-api-access-d9dx4\") pod \"multus-b5zsn\" (UID: \"759d3e52-70f2-4924-8b94-f06cf30df241\") " pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.025634 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.025575 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-56.ec2.internal" event={"ID":"dc49aa9b1d03314cbcca016a92d97157","Type":"ContainerStarted","Data":"5ee883f8324a5d743816277f9e1fbdb7d31a7f2729f997603e2e42f6975c2150"} Jul 17 21:27:41.028307 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.028239 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal" event={"ID":"d1a835e2ecd8bc0f785c433c4eda9ce2","Type":"ContainerStarted","Data":"a41667f308c8bf24298627aa22f023384bc3651defd5019dbdba58baf1256b7a"} Jul 17 21:27:41.108797 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.108762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd84q\" (UniqueName: \"kubernetes.io/projected/fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694-kube-api-access-cd84q\") pod \"node-ca-6mqj2\" (UID: \"fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694\") " pod="openshift-image-registry/node-ca-6mqj2" Jul 17 21:27:41.108797 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.108800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-run-openvswitch\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.109004 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.108818 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-sysconfig\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.109004 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.108835 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-systemd\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.109004 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.108863 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc3959d9-8af3-4c63-a611-9fb0321a7e47-agent-certs\") pod \"konnectivity-agent-bvspz\" (UID: \"cc3959d9-8af3-4c63-a611-9fb0321a7e47\") " pod="kube-system/konnectivity-agent-bvspz" Jul 17 21:27:41.109004 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.108883 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-sysctl-d\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.109004 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.108903 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gvm6\" (UniqueName: \"kubernetes.io/projected/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-kube-api-access-9gvm6\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.109004 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.108931 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-system-cni-dir\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.109004 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.108957 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-kubelet\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.109004 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.108981 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-var-lib-openvswitch\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.109004 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109006 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-ovnkube-config\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.109265 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109041 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-env-overrides\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.109265 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-sys\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.109265 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-kubernetes\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.109265 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109103 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-tuned\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.109265 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109120 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chbpv\" (UniqueName: \"kubernetes.io/projected/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-kube-api-access-chbpv\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.109265 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109135 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-slash\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.109265 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-ovnkube-script-lib\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.109265 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-var-lib-kubelet\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.109265 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109211 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694-host\") pod \"node-ca-6mqj2\" (UID: \"fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694\") " pod="openshift-image-registry/node-ca-6mqj2" Jul 17 21:27:41.109265 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.109265 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109247 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.109647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109271 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-run-ovn-kubernetes\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.109647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109291 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-cni-bin\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.109647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109306 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-run-ovn\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.109647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109302 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-run-openvswitch\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.109647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109328 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-systemd-units\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.109647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109373 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-systemd-units\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.109647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109378 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-sysconfig\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.109647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109432 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-system-cni-dir\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.109647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109447 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-systemd\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.109647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109454 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-run-systemd\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.109647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfsjf\" (UniqueName: \"kubernetes.io/projected/3e6f72d2-a524-4587-a7f5-278128245a09-kube-api-access-kfsjf\") pod \"node-resolver-4fgf5\" (UID: \"3e6f72d2-a524-4587-a7f5-278128245a09\") " pod="openshift-dns/node-resolver-4fgf5" Jul 17 21:27:41.109647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109495 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-cni-netd\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.109647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-etc-openvswitch\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.109647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109534 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-registration-dir\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.109647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109550 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc3959d9-8af3-4c63-a611-9fb0321a7e47-konnectivity-ca\") pod \"konnectivity-agent-bvspz\" (UID: \"cc3959d9-8af3-4c63-a611-9fb0321a7e47\") " pod="kube-system/konnectivity-agent-bvspz" Jul 17 21:27:41.109647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6f72d2-a524-4587-a7f5-278128245a09-tmp-dir\") pod \"node-resolver-4fgf5\" (UID: \"3e6f72d2-a524-4587-a7f5-278128245a09\") " pod="openshift-dns/node-resolver-4fgf5" Jul 17 21:27:41.109647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109595 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3ea973f-03c8-4cf1-b45b-b557aef07ef4-host-slash\") pod \"iptables-alerter-h9dwj\" (UID: \"f3ea973f-03c8-4cf1-b45b-b557aef07ef4\") " pod="openshift-network-operator/iptables-alerter-h9dwj" Jul 17 21:27:41.110454 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109610 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-os-release\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.110454 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109592 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-sysctl-d\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.110454 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-device-dir\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.110454 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109670 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-device-dir\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.110454 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109671 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svr7z\" (UniqueName: \"kubernetes.io/projected/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-kube-api-access-svr7z\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.110454 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109697 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-kubelet\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.110454 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109702 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.110454 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109723 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-var-lib-openvswitch\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.110454 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109729 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-sys-fs\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.110454 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109755 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.110454 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109784 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.110454 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109810 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-modprobe-d\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.110454 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109834 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-sysctl-conf\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.110454 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109856 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-tmp\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.110454 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109879 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-cni-binary-copy\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.110454 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109904 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-socket-dir\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.110454 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109926 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-run\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.111044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109951 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694-serviceca\") pod \"node-ca-6mqj2\" (UID: \"fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694\") " pod="openshift-image-registry/node-ca-6mqj2" Jul 17 21:27:41.111044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.109984 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-cnibin\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.111044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.110008 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-run-netns\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.111044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.110030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.111044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.110059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f3ea973f-03c8-4cf1-b45b-b557aef07ef4-iptables-alerter-script\") pod \"iptables-alerter-h9dwj\" (UID: \"f3ea973f-03c8-4cf1-b45b-b557aef07ef4\") " pod="openshift-network-operator/iptables-alerter-h9dwj" Jul 17 21:27:41.111044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.110083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg2t2\" (UniqueName: \"kubernetes.io/projected/f3ea973f-03c8-4cf1-b45b-b557aef07ef4-kube-api-access-zg2t2\") pod \"iptables-alerter-h9dwj\" (UID: \"f3ea973f-03c8-4cf1-b45b-b557aef07ef4\") " pod="openshift-network-operator/iptables-alerter-h9dwj" Jul 17 21:27:41.111044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.110110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdgbk\" (UniqueName: \"kubernetes.io/projected/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-kube-api-access-hdgbk\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.111044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.110134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-lib-modules\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.111044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.110157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-host\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.111044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.110180 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-node-log\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.111044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.110211 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-log-socket\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.111044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.110250 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-ovnkube-config\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.111044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.110262 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-ovn-node-metrics-cert\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.111044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.110287 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3e6f72d2-a524-4587-a7f5-278128245a09-hosts-file\") pod \"node-resolver-4fgf5\" (UID: \"3e6f72d2-a524-4587-a7f5-278128245a09\") " pod="openshift-dns/node-resolver-4fgf5" Jul 17 21:27:41.111044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.110437 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3e6f72d2-a524-4587-a7f5-278128245a09-hosts-file\") pod \"node-resolver-4fgf5\" (UID: \"3e6f72d2-a524-4587-a7f5-278128245a09\") " pod="openshift-dns/node-resolver-4fgf5" Jul 17 21:27:41.111044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.110604 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-env-overrides\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.111044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.110658 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-sys\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.111575 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.110726 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.111575 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.110780 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-kubernetes\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.111575 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.110836 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-sys-fs\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.111575 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.110936 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-run-ovn\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.111575 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.111168 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-slash\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.111575 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.111357 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.111575 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.111493 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.111850 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.111580 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-modprobe-d\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.111850 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.111602 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-ovnkube-script-lib\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.111850 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.111679 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-var-lib-kubelet\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.111850 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.111701 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-sysctl-conf\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.111850 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.111729 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694-host\") pod \"node-ca-6mqj2\" (UID: \"fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694\") " pod="openshift-image-registry/node-ca-6mqj2" Jul 17 21:27:41.112016 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.111853 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.112233 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.112214 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-cni-binary-copy\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.112302 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.112246 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.112302 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.112273 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-run-systemd\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.112421 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.112311 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-run-ovn-kubernetes\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.112421 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.112360 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-cni-bin\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.112421 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.112365 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-socket-dir\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.112623 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.112436 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-run\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.112623 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.112530 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-os-release\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.112733 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.112716 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6f72d2-a524-4587-a7f5-278128245a09-tmp-dir\") pod \"node-resolver-4fgf5\" (UID: \"3e6f72d2-a524-4587-a7f5-278128245a09\") " pod="openshift-dns/node-resolver-4fgf5" Jul 17 21:27:41.112893 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.112877 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc3959d9-8af3-4c63-a611-9fb0321a7e47-konnectivity-ca\") pod \"konnectivity-agent-bvspz\" (UID: \"cc3959d9-8af3-4c63-a611-9fb0321a7e47\") " pod="kube-system/konnectivity-agent-bvspz" Jul 17 21:27:41.113107 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.113081 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-etc-openvswitch\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.113169 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.113129 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694-serviceca\") pod \"node-ca-6mqj2\" (UID: \"fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694\") " pod="openshift-image-registry/node-ca-6mqj2" Jul 17 21:27:41.113169 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.113150 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-cni-netd\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.113262 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.113173 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-etc-tuned\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.113262 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.113197 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-cnibin\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.113262 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.113244 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-registration-dir\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.113453 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.113260 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.113453 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.113281 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-log-socket\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.113453 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.113299 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-lib-modules\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.113453 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.113315 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-host\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.113453 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.113356 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-node-log\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.113710 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.113627 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3ea973f-03c8-4cf1-b45b-b557aef07ef4-host-slash\") pod \"iptables-alerter-h9dwj\" (UID: \"f3ea973f-03c8-4cf1-b45b-b557aef07ef4\") " pod="openshift-network-operator/iptables-alerter-h9dwj" Jul 17 21:27:41.113710 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.113679 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-host-run-netns\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.113815 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.113745 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f3ea973f-03c8-4cf1-b45b-b557aef07ef4-iptables-alerter-script\") pod \"iptables-alerter-h9dwj\" (UID: \"f3ea973f-03c8-4cf1-b45b-b557aef07ef4\") " pod="openshift-network-operator/iptables-alerter-h9dwj" Jul 17 21:27:41.116358 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.116339 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc3959d9-8af3-4c63-a611-9fb0321a7e47-agent-certs\") pod \"konnectivity-agent-bvspz\" (UID: \"cc3959d9-8af3-4c63-a611-9fb0321a7e47\") " pod="kube-system/konnectivity-agent-bvspz" Jul 17 21:27:41.116890 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.116770 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-ovn-node-metrics-cert\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.118062 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.118011 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-tmp\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.123686 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.123660 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gvm6\" (UniqueName: \"kubernetes.io/projected/c1eeabe3-638d-4357-b6f7-6a3be3b14f7b-kube-api-access-9gvm6\") pod \"tuned-cshwz\" (UID: \"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b\") " pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.125809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.125784 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdgbk\" (UniqueName: \"kubernetes.io/projected/a63ee1de-41e9-4f10-8b76-994e76ac3a7c-kube-api-access-hdgbk\") pod \"aws-ebs-csi-driver-node-hlzdw\" (UID: \"a63ee1de-41e9-4f10-8b76-994e76ac3a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.126449 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.126430 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfsjf\" (UniqueName: \"kubernetes.io/projected/3e6f72d2-a524-4587-a7f5-278128245a09-kube-api-access-kfsjf\") pod \"node-resolver-4fgf5\" (UID: \"3e6f72d2-a524-4587-a7f5-278128245a09\") " pod="openshift-dns/node-resolver-4fgf5" Jul 17 21:27:41.127169 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.127095 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chbpv\" (UniqueName: \"kubernetes.io/projected/ba80c237-8ba4-4bdc-9592-c8fb5bf8d912-kube-api-access-chbpv\") pod \"ovnkube-node-rhkgr\" (UID: \"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912\") " pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.127719 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.127674 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd84q\" (UniqueName: \"kubernetes.io/projected/fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694-kube-api-access-cd84q\") pod \"node-ca-6mqj2\" (UID: \"fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694\") " pod="openshift-image-registry/node-ca-6mqj2" Jul 17 21:27:41.127869 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.127749 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svr7z\" (UniqueName: \"kubernetes.io/projected/21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66-kube-api-access-svr7z\") pod \"multus-additional-cni-plugins-qlgx8\" (UID: \"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66\") " pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.129932 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.129912 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg2t2\" (UniqueName: \"kubernetes.io/projected/f3ea973f-03c8-4cf1-b45b-b557aef07ef4-kube-api-access-zg2t2\") pod \"iptables-alerter-h9dwj\" (UID: \"f3ea973f-03c8-4cf1-b45b-b557aef07ef4\") " pod="openshift-network-operator/iptables-alerter-h9dwj" Jul 17 21:27:41.130932 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.130915 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:27:41.205336 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.205298 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b5zsn" Jul 17 21:27:41.218282 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.218259 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h9dwj" Jul 17 21:27:41.226158 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.226138 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:27:41.238989 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.238963 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" Jul 17 21:27:41.249740 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.249722 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bvspz" Jul 17 21:27:41.259524 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.259499 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cshwz" Jul 17 21:27:41.265185 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.265162 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4fgf5" Jul 17 21:27:41.272782 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.272760 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6mqj2" Jul 17 21:27:41.279473 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.279455 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qlgx8" Jul 17 21:27:41.512991 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.512896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret\") pod \"global-pull-secret-syncer-fl9qj\" (UID: \"f44bc251-268b-42f5-b9d3-a742151884f9\") " pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:41.512991 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.512960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs\") pod \"network-metrics-daemon-d2x4m\" (UID: \"896ed8fb-96fc-4834-83a6-807a16a340b3\") " pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:41.513195 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:41.513055 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:41.513195 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:41.513073 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:41.513195 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:41.513118 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret podName:f44bc251-268b-42f5-b9d3-a742151884f9 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:42.513097793 +0000 UTC m=+3.934817010 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret") pod "global-pull-secret-syncer-fl9qj" (UID: "f44bc251-268b-42f5-b9d3-a742151884f9") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:41.513195 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:41.513139 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs podName:896ed8fb-96fc-4834-83a6-807a16a340b3 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:42.513130543 +0000 UTC m=+3.934849755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs") pod "network-metrics-daemon-d2x4m" (UID: "896ed8fb-96fc-4834-83a6-807a16a340b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:41.614244 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.614206 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtgm2\" (UniqueName: \"kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2\") pod \"network-check-target-n7c88\" (UID: \"c85213e4-c44c-418e-ae68-c74526ae6fa5\") " pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:27:41.614455 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:41.614414 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:27:41.614455 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:41.614441 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:27:41.614569 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:41.614457 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mtgm2 for pod openshift-network-diagnostics/network-check-target-n7c88: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:41.614569 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:41.614517 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2 podName:c85213e4-c44c-418e-ae68-c74526ae6fa5 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:42.614501701 +0000 UTC m=+4.036220915 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtgm2" (UniqueName: "kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2") pod "network-check-target-n7c88" (UID: "c85213e4-c44c-418e-ae68-c74526ae6fa5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:41.928833 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.928792 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 21:22:40 +0000 UTC" deadline="2028-04-25 00:30:45.469643254 +0000 UTC" Jul 17 21:27:41.928833 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.928830 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15531h3m3.540816711s" Jul 17 21:27:41.987893 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:41.987854 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:27:42.018459 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:42.018416 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:42.018622 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:42.018541 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:27:42.083795 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:42.083759 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba80c237_8ba4_4bdc_9592_c8fb5bf8d912.slice/crio-c2cb56946cb5ab981c518234827fd6ca5189ffdfa681aa1f605152a141ec30c4 WatchSource:0}: Error finding container c2cb56946cb5ab981c518234827fd6ca5189ffdfa681aa1f605152a141ec30c4: Status 404 returned error can't find the container with id c2cb56946cb5ab981c518234827fd6ca5189ffdfa681aa1f605152a141ec30c4 Jul 17 21:27:42.089713 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:42.089685 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8ddec3_581d_4bd5_bfa6_0fd4d60b1694.slice/crio-1559cfd4e1ece3e57264c0addf21670e76504496e412b586af24df118ff6fc55 WatchSource:0}: Error finding container 1559cfd4e1ece3e57264c0addf21670e76504496e412b586af24df118ff6fc55: Status 404 returned error can't find the container with id 1559cfd4e1ece3e57264c0addf21670e76504496e412b586af24df118ff6fc55 Jul 17 21:27:42.090293 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:42.090266 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ea973f_03c8_4cf1_b45b_b557aef07ef4.slice/crio-d0dc44ad5a82555677ca5cf8113fafdc064991cf4e434ce3ed117a0e838ecbd9 WatchSource:0}: Error finding container d0dc44ad5a82555677ca5cf8113fafdc064991cf4e434ce3ed117a0e838ecbd9: Status 404 returned error can't find the container with id d0dc44ad5a82555677ca5cf8113fafdc064991cf4e434ce3ed117a0e838ecbd9 Jul 17 21:27:42.091274 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:27:42.091078 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1eeabe3_638d_4357_b6f7_6a3be3b14f7b.slice/crio-ada6df6d766801ac31a27078de3f8b409425d70a940ef59dbf645adbdd141f14 WatchSource:0}: Error finding container ada6df6d766801ac31a27078de3f8b409425d70a940ef59dbf645adbdd141f14: Status 404 returned error can't find the container with id ada6df6d766801ac31a27078de3f8b409425d70a940ef59dbf645adbdd141f14 Jul 17 21:27:42.520225 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:42.520095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs\") pod \"network-metrics-daemon-d2x4m\" (UID: \"896ed8fb-96fc-4834-83a6-807a16a340b3\") " pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:42.520225 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:42.520182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret\") pod \"global-pull-secret-syncer-fl9qj\" (UID: \"f44bc251-268b-42f5-b9d3-a742151884f9\") " pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:42.520541 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:42.520265 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:42.520541 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:42.520303 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:42.520541 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:42.520335 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs podName:896ed8fb-96fc-4834-83a6-807a16a340b3 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:44.520315798 +0000 UTC m=+5.942035011 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs") pod "network-metrics-daemon-d2x4m" (UID: "896ed8fb-96fc-4834-83a6-807a16a340b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:42.520541 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:42.520367 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret podName:f44bc251-268b-42f5-b9d3-a742151884f9 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:44.52034751 +0000 UTC m=+5.942066728 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret") pod "global-pull-secret-syncer-fl9qj" (UID: "f44bc251-268b-42f5-b9d3-a742151884f9") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:42.621246 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:42.621211 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtgm2\" (UniqueName: \"kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2\") pod \"network-check-target-n7c88\" (UID: \"c85213e4-c44c-418e-ae68-c74526ae6fa5\") " pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:27:42.621455 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:42.621343 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:27:42.621455 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:42.621358 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:27:42.621455 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:42.621371 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mtgm2 for pod openshift-network-diagnostics/network-check-target-n7c88: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:42.621455 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:42.621434 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2 podName:c85213e4-c44c-418e-ae68-c74526ae6fa5 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:44.621419286 +0000 UTC m=+6.043138496 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtgm2" (UniqueName: "kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2") pod "network-check-target-n7c88" (UID: "c85213e4-c44c-418e-ae68-c74526ae6fa5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:42.929202 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:42.929086 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 21:22:40 +0000 UTC" deadline="2028-02-06 12:15:13.950345836 +0000 UTC" Jul 17 21:27:42.929202 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:42.929148 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13646h47m31.02120261s" Jul 17 21:27:43.022778 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:43.022080 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:43.022778 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:43.022227 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:27:43.022778 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:43.022636 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:27:43.022778 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:43.022722 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:27:43.037454 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:43.037384 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4fgf5" event={"ID":"3e6f72d2-a524-4587-a7f5-278128245a09","Type":"ContainerStarted","Data":"d45cad799f997beab48f5e1c20aa1744db116cd7e40dcb2cbe0439f43b0c2be6"} Jul 17 21:27:43.040093 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:43.040040 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bvspz" event={"ID":"cc3959d9-8af3-4c63-a611-9fb0321a7e47","Type":"ContainerStarted","Data":"ff1242f51fe99e722c6040f2b0afe6916ed20a96face035cf888000fc37cc0d4"} Jul 17 21:27:43.048324 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:43.048142 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cshwz" event={"ID":"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b","Type":"ContainerStarted","Data":"ada6df6d766801ac31a27078de3f8b409425d70a940ef59dbf645adbdd141f14"} Jul 17 21:27:43.051564 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:43.051492 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h9dwj" event={"ID":"f3ea973f-03c8-4cf1-b45b-b557aef07ef4","Type":"ContainerStarted","Data":"d0dc44ad5a82555677ca5cf8113fafdc064991cf4e434ce3ed117a0e838ecbd9"} Jul 17 21:27:43.062779 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:43.062745 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6mqj2" event={"ID":"fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694","Type":"ContainerStarted","Data":"1559cfd4e1ece3e57264c0addf21670e76504496e412b586af24df118ff6fc55"} Jul 17 21:27:43.073955 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:43.073920 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-56.ec2.internal" event={"ID":"dc49aa9b1d03314cbcca016a92d97157","Type":"ContainerStarted","Data":"13d2c8d68d2eda2dd2490d6fe15c2e931cfb3d438693abb2dc995185fbe63b70"} Jul 17 21:27:43.078830 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:43.078685 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlgx8" event={"ID":"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66","Type":"ContainerStarted","Data":"8c025af5038ea925918b9bbf98492ea6de02351289fb043a20930e17f5486fd2"} Jul 17 21:27:43.084054 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:43.084023 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" event={"ID":"a63ee1de-41e9-4f10-8b76-994e76ac3a7c","Type":"ContainerStarted","Data":"1e47e8571ffd79afc44d6e839344fe10562aaedc501055456cac9002ac122244"} Jul 17 21:27:43.100003 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:43.099969 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" event={"ID":"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912","Type":"ContainerStarted","Data":"c2cb56946cb5ab981c518234827fd6ca5189ffdfa681aa1f605152a141ec30c4"} Jul 17 21:27:43.130663 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:43.130490 2580 generic.go:356] "Generic (PLEG): container finished" podID="d1a835e2ecd8bc0f785c433c4eda9ce2" containerID="1fd177c764b07fbb356e5219a6248e7705a1211bebfed231e538e9032338c987" exitCode=0 Jul 17 21:27:43.130663 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:43.130593 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal" event={"ID":"d1a835e2ecd8bc0f785c433c4eda9ce2","Type":"ContainerDied","Data":"1fd177c764b07fbb356e5219a6248e7705a1211bebfed231e538e9032338c987"} Jul 17 21:27:43.139812 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:43.139780 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b5zsn" event={"ID":"759d3e52-70f2-4924-8b94-f06cf30df241","Type":"ContainerStarted","Data":"b357f4cdb1e2f8ddd25db66297602a82309d5637604656bc866a10ee36647a25"} Jul 17 21:27:43.146950 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:43.146899 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-56.ec2.internal" podStartSLOduration=3.1468788229999998 podStartE2EDuration="3.146878823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:27:43.090646244 +0000 UTC m=+4.512365497" watchObservedRunningTime="2026-07-17 21:27:43.146878823 +0000 UTC m=+4.568598058" Jul 17 21:27:44.019067 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:44.018531 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:44.019067 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:44.018669 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:27:44.159767 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:44.159730 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal" event={"ID":"d1a835e2ecd8bc0f785c433c4eda9ce2","Type":"ContainerStarted","Data":"2e59f40cc5e50f69c36a8e26ab1b1d79091ad85860b97eb3b9f8a0dde06d676c"} Jul 17 21:27:44.539337 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:44.539297 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret\") pod \"global-pull-secret-syncer-fl9qj\" (UID: \"f44bc251-268b-42f5-b9d3-a742151884f9\") " pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:44.539547 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:44.539366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs\") pod \"network-metrics-daemon-d2x4m\" (UID: \"896ed8fb-96fc-4834-83a6-807a16a340b3\") " pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:44.539547 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:44.539517 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:44.539655 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:44.539583 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs podName:896ed8fb-96fc-4834-83a6-807a16a340b3 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:48.539564489 +0000 UTC m=+9.961283713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs") pod "network-metrics-daemon-d2x4m" (UID: "896ed8fb-96fc-4834-83a6-807a16a340b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:44.540008 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:44.539983 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:44.540107 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:44.540046 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret podName:f44bc251-268b-42f5-b9d3-a742151884f9 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:48.540030193 +0000 UTC m=+9.961749409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret") pod "global-pull-secret-syncer-fl9qj" (UID: "f44bc251-268b-42f5-b9d3-a742151884f9") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:44.640727 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:44.640061 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtgm2\" (UniqueName: \"kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2\") pod \"network-check-target-n7c88\" (UID: \"c85213e4-c44c-418e-ae68-c74526ae6fa5\") " pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:27:44.640727 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:44.640269 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:27:44.640727 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:44.640289 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:27:44.640727 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:44.640301 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mtgm2 for pod openshift-network-diagnostics/network-check-target-n7c88: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:44.640727 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:44.640366 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2 podName:c85213e4-c44c-418e-ae68-c74526ae6fa5 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:48.640347873 +0000 UTC m=+10.062067088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtgm2" (UniqueName: "kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2") pod "network-check-target-n7c88" (UID: "c85213e4-c44c-418e-ae68-c74526ae6fa5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:45.018624 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:45.018546 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:45.018778 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:45.018696 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:27:45.019420 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:45.019377 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:27:45.019799 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:45.019505 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:27:46.018532 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:46.018501 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:46.018733 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:46.018630 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:27:47.017849 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:47.017813 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:47.018286 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:47.017946 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:27:47.018286 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:47.018276 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:27:47.018403 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:47.018370 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:27:48.018674 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:48.018217 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:48.018674 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:48.018346 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:27:48.582899 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:48.582854 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret\") pod \"global-pull-secret-syncer-fl9qj\" (UID: \"f44bc251-268b-42f5-b9d3-a742151884f9\") " pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:48.583091 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:48.582925 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs\") pod \"network-metrics-daemon-d2x4m\" (UID: \"896ed8fb-96fc-4834-83a6-807a16a340b3\") " pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:48.583091 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:48.583078 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:48.583179 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:48.583142 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs podName:896ed8fb-96fc-4834-83a6-807a16a340b3 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:56.583121256 +0000 UTC m=+18.004840472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs") pod "network-metrics-daemon-d2x4m" (UID: "896ed8fb-96fc-4834-83a6-807a16a340b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:48.583558 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:48.583525 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:48.583686 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:48.583588 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret podName:f44bc251-268b-42f5-b9d3-a742151884f9 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:56.583574373 +0000 UTC m=+18.005293585 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret") pod "global-pull-secret-syncer-fl9qj" (UID: "f44bc251-268b-42f5-b9d3-a742151884f9") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:48.683406 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:48.683354 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtgm2\" (UniqueName: \"kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2\") pod \"network-check-target-n7c88\" (UID: \"c85213e4-c44c-418e-ae68-c74526ae6fa5\") " pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:27:48.683570 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:48.683544 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:27:48.683647 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:48.683576 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:27:48.683647 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:48.683594 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mtgm2 for pod openshift-network-diagnostics/network-check-target-n7c88: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:48.683745 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:48.683665 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2 podName:c85213e4-c44c-418e-ae68-c74526ae6fa5 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:56.683644125 +0000 UTC m=+18.105363350 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtgm2" (UniqueName: "kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2") pod "network-check-target-n7c88" (UID: "c85213e4-c44c-418e-ae68-c74526ae6fa5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:49.026750 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:49.023718 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:27:49.026750 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:49.023945 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:27:49.026750 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:49.024328 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:49.026750 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:49.024438 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:27:50.018069 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:50.017575 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:50.018069 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:50.017711 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:27:51.021517 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:51.021075 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:27:51.021517 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:51.021193 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:27:51.021517 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:51.021467 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:51.022092 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:51.021601 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:27:52.017996 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:52.017910 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:52.018186 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:52.018056 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:27:53.018578 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:53.018541 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:27:53.019032 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:53.018686 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:27:53.019032 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:53.018541 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:53.019032 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:53.018808 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:27:54.017771 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:54.017732 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:54.017946 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:54.017866 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:27:55.021013 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:55.020984 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:55.021512 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:55.021110 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:27:55.021512 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:55.021460 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:27:55.021613 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:55.021523 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:27:56.018140 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:56.018102 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:56.018321 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:56.018237 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:27:56.647737 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:56.647700 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret\") pod \"global-pull-secret-syncer-fl9qj\" (UID: \"f44bc251-268b-42f5-b9d3-a742151884f9\") " pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:56.648139 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:56.647760 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs\") pod \"network-metrics-daemon-d2x4m\" (UID: \"896ed8fb-96fc-4834-83a6-807a16a340b3\") " pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:56.648139 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:56.647847 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:56.648139 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:56.647908 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:56.648139 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:56.647912 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret podName:f44bc251-268b-42f5-b9d3-a742151884f9 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:12.64789591 +0000 UTC m=+34.069615121 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret") pod "global-pull-secret-syncer-fl9qj" (UID: "f44bc251-268b-42f5-b9d3-a742151884f9") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:56.648139 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:56.647970 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs podName:896ed8fb-96fc-4834-83a6-807a16a340b3 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:12.647953058 +0000 UTC m=+34.069672276 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs") pod "network-metrics-daemon-d2x4m" (UID: "896ed8fb-96fc-4834-83a6-807a16a340b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:56.748504 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:56.748461 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtgm2\" (UniqueName: \"kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2\") pod \"network-check-target-n7c88\" (UID: \"c85213e4-c44c-418e-ae68-c74526ae6fa5\") " pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:27:56.748676 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:56.748652 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:27:56.748740 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:56.748703 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:27:56.748740 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:56.748719 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mtgm2 for pod openshift-network-diagnostics/network-check-target-n7c88: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:56.748817 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:56.748789 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2 podName:c85213e4-c44c-418e-ae68-c74526ae6fa5 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:12.74876831 +0000 UTC m=+34.170487526 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtgm2" (UniqueName: "kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2") pod "network-check-target-n7c88" (UID: "c85213e4-c44c-418e-ae68-c74526ae6fa5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:57.018207 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:57.018122 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:57.018423 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:57.018269 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:27:57.018423 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:57.018298 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:27:57.018593 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:57.018437 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:27:58.018443 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:58.018383 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:27:58.018856 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:58.018530 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:27:59.019204 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:59.019164 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:27:59.019692 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:59.019279 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:27:59.019692 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:27:59.019331 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:27:59.019692 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:27:59.019438 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:28:00.018375 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:00.018336 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:00.018582 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:00.018484 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:28:01.019671 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:01.019647 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:01.020271 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:01.020243 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:28:01.025213 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:01.023510 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:01.025559 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:01.025527 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:28:01.197418 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:01.197371 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlgx8" event={"ID":"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66","Type":"ContainerStarted","Data":"1014a75ab7e42e4332f5f4dea67c913538b26ea25e654d8d56d88a25b83539b8"} Jul 17 21:28:01.201771 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:01.201703 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" event={"ID":"a63ee1de-41e9-4f10-8b76-994e76ac3a7c","Type":"ContainerStarted","Data":"2e60e99bd80806d24e7c932c364e29eed302ebd9f986df454269c1965d9b2272"} Jul 17 21:28:01.209311 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:01.208818 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4fgf5" event={"ID":"3e6f72d2-a524-4587-a7f5-278128245a09","Type":"ContainerStarted","Data":"b38dd909b888ac9dfcf6a589dfa8bc6a6c8ac78021d29f8d21d48cb2bbd5572c"} Jul 17 21:28:01.215407 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:01.215361 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bvspz" event={"ID":"cc3959d9-8af3-4c63-a611-9fb0321a7e47","Type":"ContainerStarted","Data":"f83e3a601aef9213214c360b1f8a4f3a7c44620601c05aca7fcc74d8c3c2c357"} Jul 17 21:28:01.220859 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:01.220200 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-56.ec2.internal" podStartSLOduration=21.220186086 podStartE2EDuration="21.220186086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:27:44.181202977 +0000 UTC m=+5.602922272" watchObservedRunningTime="2026-07-17 21:28:01.220186086 +0000 UTC m=+22.641905318" Jul 17 21:28:01.233061 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:01.233010 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4fgf5" podStartSLOduration=3.393157544 podStartE2EDuration="22.232991774s" totalImagesPullingTime="18.83983423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:01.232826288 +0000 UTC m=+22.654545540" watchObservedRunningTime="2026-07-17 21:28:01.232991774 +0000 UTC m=+22.654711008" Jul 17 21:28:01.248954 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:01.248906 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bvspz" podStartSLOduration=3.385603235 podStartE2EDuration="22.248893327s" totalImagesPullingTime="18.863290092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:01.248306397 +0000 UTC m=+22.670025656" watchObservedRunningTime="2026-07-17 21:28:01.248893327 +0000 UTC m=+22.670612561" Jul 17 21:28:01.390874 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:01.390836 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:01.406723 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:01.406700 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bvspz" Jul 17 21:28:01.407185 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:01.407166 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bvspz" Jul 17 21:28:01.603550 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:01.603517 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:01.703368 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:01.703199 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:01.924267 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:01.924233 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:02.018281 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.018249 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:02.018449 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:02.018410 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:28:02.027753 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.027721 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:02.148031 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.147999 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bvspz" Jul 17 21:28:02.148523 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.148507 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bvspz" Jul 17 21:28:02.219313 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.219279 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6mqj2" event={"ID":"fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694","Type":"ContainerStarted","Data":"5f50c97af83b230eb6f8734c2603f715d3bdfc01ff0a76db96c918564652d21f"} Jul 17 21:28:02.220746 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.220719 2580 generic.go:356] "Generic (PLEG): container finished" podID="21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66" containerID="1014a75ab7e42e4332f5f4dea67c913538b26ea25e654d8d56d88a25b83539b8" exitCode=0 Jul 17 21:28:02.220863 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.220811 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlgx8" event={"ID":"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66","Type":"ContainerDied","Data":"1014a75ab7e42e4332f5f4dea67c913538b26ea25e654d8d56d88a25b83539b8"} Jul 17 21:28:02.223816 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.223788 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 21:28:02.224145 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.224114 2580 generic.go:356] "Generic (PLEG): container finished" podID="ba80c237-8ba4-4bdc-9592-c8fb5bf8d912" containerID="2a212fab3a5ec27828ba5749b98f092908e76a9c7b00b2cf27a8b24b0d5c6b3d" exitCode=1 Jul 17 21:28:02.224250 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.224183 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" event={"ID":"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912","Type":"ContainerStarted","Data":"d81fd1a2cf0b5662885eae39299aa147677dc076057e74ea55777fd2d21165b6"} Jul 17 21:28:02.224250 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.224201 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:02.224250 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.224245 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" event={"ID":"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912","Type":"ContainerStarted","Data":"395529e6237095c5e966ba5a22ae03390e2babf008a0a2416ba8ba37127740d0"} Jul 17 21:28:02.224441 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.224289 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" event={"ID":"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912","Type":"ContainerStarted","Data":"f638a417cfe934c8b93b30d8406364e727d3f79ddf045e897bfcf31792ab8090"} Jul 17 21:28:02.224441 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.224320 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" event={"ID":"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912","Type":"ContainerStarted","Data":"d7e894a8abba2b404feded5901c1927b7e937b4762120b706234ddf847b79cb8"} Jul 17 21:28:02.224441 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.224407 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" event={"ID":"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912","Type":"ContainerDied","Data":"2a212fab3a5ec27828ba5749b98f092908e76a9c7b00b2cf27a8b24b0d5c6b3d"} Jul 17 21:28:02.224556 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.224458 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" event={"ID":"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912","Type":"ContainerStarted","Data":"a7ac13a0331654153509cc2df46a23fa5328fc41a0082c0065d62e3a4fb315e6"} Jul 17 21:28:02.225553 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.225534 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b5zsn" event={"ID":"759d3e52-70f2-4924-8b94-f06cf30df241","Type":"ContainerStarted","Data":"8ef22b76f88dc21f523fc25201242ada294e756eddaea1746088d55883e629e6"} Jul 17 21:28:02.227097 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.227078 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cshwz" event={"ID":"c1eeabe3-638d-4357-b6f7-6a3be3b14f7b","Type":"ContainerStarted","Data":"0e5b57fab9dffe9c9a07c91215a64a9dd642fb2765bf2af55c9cdd4ace9ef728"} Jul 17 21:28:02.249629 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.249588 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6mqj2" podStartSLOduration=4.361844335 podStartE2EDuration="23.249572178s" totalImagesPullingTime="18.887727843s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:02.233211107 +0000 UTC m=+23.654930343" watchObservedRunningTime="2026-07-17 21:28:02.249572178 +0000 UTC m=+23.671291411" Jul 17 21:28:02.249849 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.249827 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cshwz" podStartSLOduration=4.368294068 podStartE2EDuration="23.249821879s" totalImagesPullingTime="18.881527811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:02.249144619 +0000 UTC m=+23.670863853" watchObservedRunningTime="2026-07-17 21:28:02.249821879 +0000 UTC m=+23.671541112" Jul 17 21:28:02.265496 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.265415 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-b5zsn" podStartSLOduration=4.360151556 podStartE2EDuration="23.265377958s" totalImagesPullingTime="18.905226402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:02.264903184 +0000 UTC m=+23.686622418" watchObservedRunningTime="2026-07-17 21:28:02.265377958 +0000 UTC m=+23.687097193" Jul 17 21:28:02.359333 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.359305 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:02.414887 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.414853 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:02.560931 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.560905 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:02.581293 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.581270 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 17 21:28:02.954205 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.954045 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T21:28:02.581287777Z","UUID":"85e0791a-3d52-4762-af90-818c3765ce0b","Handler":null,"Name":"","Endpoint":""} Jul 17 21:28:02.955670 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.955646 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 17 21:28:02.955821 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:02.955684 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 17 21:28:03.018529 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:03.018486 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:03.018529 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:03.018520 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:03.018741 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:03.018610 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:28:03.019017 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:03.018993 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:28:03.022138 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:03.022108 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:03.116127 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:03.116093 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:03.219678 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:03.219580 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:03.231045 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:03.231012 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" event={"ID":"a63ee1de-41e9-4f10-8b76-994e76ac3a7c","Type":"ContainerStarted","Data":"1ba77f99056fd2996b500405e331f72f6d234129c78498da930cb94706413eaa"} Jul 17 21:28:03.232674 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:03.232646 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h9dwj" event={"ID":"f3ea973f-03c8-4cf1-b45b-b557aef07ef4","Type":"ContainerStarted","Data":"059541640bfa3b665f980b39f2e2f069820cd60a91fdcf665527ff7ad8e8b666"} Jul 17 21:28:03.247206 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:03.247168 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h9dwj" podStartSLOduration=5.357614947 podStartE2EDuration="24.247153801s" totalImagesPullingTime="18.889538854s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:03.247047485 +0000 UTC m=+24.668766719" watchObservedRunningTime="2026-07-17 21:28:03.247153801 +0000 UTC m=+24.668873012" Jul 17 21:28:03.427002 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:03.426967 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:03.504738 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:03.504645 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:03.590963 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:03.590938 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:03.923711 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:03.923616 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:03.976576 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:03.976541 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:04.017805 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:04.017783 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:04.017914 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:04.017872 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:28:04.030459 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:04.030440 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:04.132165 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:04.132132 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:04.237985 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:04.237889 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" event={"ID":"a63ee1de-41e9-4f10-8b76-994e76ac3a7c","Type":"ContainerStarted","Data":"1e37b7e4bef30fee7476d0eebd9068c160e40c615105d9c43c8540d42f2027f9"} Jul 17 21:28:04.241594 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:04.241568 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 21:28:04.242174 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:04.242151 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" event={"ID":"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912","Type":"ContainerStarted","Data":"842ce7c65b659804eab56a3f8ce02dc0fc0454b0387d37e3d6f4360d05bcf6ed"} Jul 17 21:28:04.256943 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:04.256891 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlzdw" podStartSLOduration=4.152402941 podStartE2EDuration="25.256874745s" totalImagesPullingTime="21.104471804s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:04.2566784 +0000 UTC m=+25.678397633" watchObservedRunningTime="2026-07-17 21:28:04.256874745 +0000 UTC m=+25.678594000" Jul 17 21:28:05.017569 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:05.017535 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:05.017742 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:05.017650 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:28:05.017742 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:05.017704 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:05.017819 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:05.017789 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:28:05.738309 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:05.738270 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:05.910909 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:05.910875 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:06.011852 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:06.011819 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:06.017858 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:06.017833 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:06.018007 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:06.017942 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:28:06.788870 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:06.788832 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:07.018183 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:07.018024 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:07.018309 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:07.018100 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:07.018309 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:07.018284 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:28:07.018412 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:07.018373 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:28:07.249244 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:07.249206 2580 generic.go:356] "Generic (PLEG): container finished" podID="21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66" containerID="6d652bbd1cfc01fe98b4e4392b5ceda959199a12417d34aab1181f1c5405c406" exitCode=0 Jul 17 21:28:07.249421 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:07.249291 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlgx8" event={"ID":"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66","Type":"ContainerDied","Data":"6d652bbd1cfc01fe98b4e4392b5ceda959199a12417d34aab1181f1c5405c406"} Jul 17 21:28:07.253003 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:07.252985 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 21:28:07.253350 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:07.253325 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" event={"ID":"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912","Type":"ContainerStarted","Data":"e07a69d5d430189952d9b89b7cb706e3aa5f06adc2076cd6f0086a8338d07768"} Jul 17 21:28:07.253648 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:07.253626 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:28:07.253773 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:07.253757 2580 scope.go:117] "RemoveContainer" containerID="2a212fab3a5ec27828ba5749b98f092908e76a9c7b00b2cf27a8b24b0d5c6b3d" Jul 17 21:28:07.269662 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:07.269643 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:28:07.765422 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:07.765376 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:07.970285 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:07.970250 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:08.018102 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:08.018080 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:08.018195 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:08.018177 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:28:08.072034 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:08.072008 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:08.171340 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:08.171305 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:08.257860 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:08.257828 2580 generic.go:356] "Generic (PLEG): container finished" podID="21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66" containerID="123f7ec98d08439d75fc4aefdb2b2c9e5fc716b69305160dc7814e649c7fe8d0" exitCode=0 Jul 17 21:28:08.258015 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:08.257916 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlgx8" event={"ID":"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66","Type":"ContainerDied","Data":"123f7ec98d08439d75fc4aefdb2b2c9e5fc716b69305160dc7814e649c7fe8d0"} Jul 17 21:28:08.261890 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:08.261872 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 21:28:08.262240 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:08.262221 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" event={"ID":"ba80c237-8ba4-4bdc-9592-c8fb5bf8d912","Type":"ContainerStarted","Data":"fe158bd839c6d37f35dbc0023ebf6cc2018400bfc396a13ab822f0f9d20ac3e8"} Jul 17 21:28:08.262404 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:08.262376 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:28:08.262455 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:08.262447 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:28:08.270306 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:08.270252 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:08.277814 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:08.277794 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:28:08.327923 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:08.327875 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" podStartSLOduration=10.177990354 podStartE2EDuration="29.327859731s" totalImagesPullingTime="19.149869377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:08.32734194 +0000 UTC m=+29.749061174" watchObservedRunningTime="2026-07-17 21:28:08.327859731 +0000 UTC m=+29.749578964" Jul 17 21:28:08.654215 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:08.654109 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:09.018837 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:09.018809 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:09.019188 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:09.018916 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:28:09.019188 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:09.018966 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:09.019188 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:09.019047 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:28:09.267341 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:09.267311 2580 generic.go:356] "Generic (PLEG): container finished" podID="21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66" containerID="800ecd823a2ead1861dd0bcb3fc90a8e384cf7dc1d625d379effe3cd1ecb6761" exitCode=0 Jul 17 21:28:09.267550 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:09.267414 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlgx8" event={"ID":"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66","Type":"ContainerDied","Data":"800ecd823a2ead1861dd0bcb3fc90a8e384cf7dc1d625d379effe3cd1ecb6761"} Jul 17 21:28:09.457079 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:09.457051 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:09.668308 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:09.668231 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:10.018586 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:10.018550 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:10.018753 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:10.018659 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:28:10.056298 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:10.056267 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:10.262660 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:10.262628 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:10.361703 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:10.361623 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:10.469562 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:10.469533 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:10.656778 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:10.656644 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:11.017699 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:11.017666 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:11.017855 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:11.017819 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:28:11.017923 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:11.017858 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:11.018007 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:11.017976 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:28:11.455471 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:11.455375 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:11.526676 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:11.526642 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:11.856921 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:11.856889 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:12.018204 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:12.018174 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:12.018373 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:12.018288 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:28:12.055096 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:12.055054 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:12.260090 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:12.260007 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:12.360569 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:12.360532 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:12.651847 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:12.651816 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:12.673912 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:12.673884 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret\") pod \"global-pull-secret-syncer-fl9qj\" (UID: \"f44bc251-268b-42f5-b9d3-a742151884f9\") " pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:12.674099 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:12.673937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs\") pod \"network-metrics-daemon-d2x4m\" (UID: \"896ed8fb-96fc-4834-83a6-807a16a340b3\") " pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:12.674099 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:12.674045 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:28:12.674099 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:12.674067 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:28:12.674224 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:12.674108 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret podName:f44bc251-268b-42f5-b9d3-a742151884f9 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:44.674091908 +0000 UTC m=+66.095811131 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret") pod "global-pull-secret-syncer-fl9qj" (UID: "f44bc251-268b-42f5-b9d3-a742151884f9") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:28:12.674224 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:12.674123 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs podName:896ed8fb-96fc-4834-83a6-807a16a340b3 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:44.674116643 +0000 UTC m=+66.095835854 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs") pod "network-metrics-daemon-d2x4m" (UID: "896ed8fb-96fc-4834-83a6-807a16a340b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:28:12.775028 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:12.774991 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtgm2\" (UniqueName: \"kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2\") pod \"network-check-target-n7c88\" (UID: \"c85213e4-c44c-418e-ae68-c74526ae6fa5\") " pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:12.775192 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:12.775131 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:28:12.775192 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:12.775150 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:28:12.775192 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:12.775158 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mtgm2 for pod openshift-network-diagnostics/network-check-target-n7c88: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:28:12.775316 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:12.775212 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2 podName:c85213e4-c44c-418e-ae68-c74526ae6fa5 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:44.775193819 +0000 UTC m=+66.196913033 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtgm2" (UniqueName: "kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2") pod "network-check-target-n7c88" (UID: "c85213e4-c44c-418e-ae68-c74526ae6fa5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:28:12.910856 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:12.910774 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:13.010987 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:13.010952 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:13.018381 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:13.018359 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:13.018514 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:13.018359 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:13.018563 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:13.018512 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:28:13.018563 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:13.018535 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:28:13.111819 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:13.111783 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:13.211422 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:13.211327 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:13.313432 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:13.313376 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:13.419804 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:13.419511 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:13.590198 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:13.590155 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:13.664788 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:13.664754 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:13.959557 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:13.959471 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:14.013671 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:14.013629 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:14.018575 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:14.018546 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:14.018712 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:14.018652 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:28:14.072277 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:14.072234 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:14.130904 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:14.130872 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:14.261834 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:14.261773 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:14.361776 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:14.361736 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:14.559099 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:14.559065 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:14.856292 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:14.856209 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:15.017763 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:15.017727 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:15.017940 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:15.017842 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:28:15.017940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:15.017902 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:15.018044 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:15.018021 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:28:15.266640 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:15.266611 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:15.282849 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:15.282819 2580 generic.go:356] "Generic (PLEG): container finished" podID="21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66" containerID="7dc9377926957e36630c8c697ce2ca10b07b474dd9a71e412ee0e6d29214d1b1" exitCode=0 Jul 17 21:28:15.282994 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:15.282881 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlgx8" event={"ID":"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66","Type":"ContainerDied","Data":"7dc9377926957e36630c8c697ce2ca10b07b474dd9a71e412ee0e6d29214d1b1"} Jul 17 21:28:15.388532 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:15.388498 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:15.488572 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:15.488542 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:15.547616 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:15.547536 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:15.853738 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:15.853646 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:16.017702 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:16.017671 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:16.018073 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:16.017770 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:28:16.288073 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:16.288036 2580 generic.go:356] "Generic (PLEG): container finished" podID="21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66" containerID="d937d590b4e224456c7f5e503237920b1a5adbcd6803e3d935c01b39f7ac0765" exitCode=0 Jul 17 21:28:16.288258 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:16.288078 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlgx8" event={"ID":"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66","Type":"ContainerDied","Data":"d937d590b4e224456c7f5e503237920b1a5adbcd6803e3d935c01b39f7ac0765"} Jul 17 21:28:16.463280 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:16.463054 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:16.856676 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:16.856645 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:17.017609 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:17.017580 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:17.017785 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:17.017624 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:17.017785 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:17.017685 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:28:17.017785 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:17.017742 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:28:17.063950 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:17.063920 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:17.293591 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:17.293555 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlgx8" event={"ID":"21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66","Type":"ContainerStarted","Data":"2461a623a6994e9735332695409b1c215fe27d52d3963d6fd353b14c62c6a3ce"} Jul 17 21:28:17.315381 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:17.315333 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qlgx8" podStartSLOduration=8.067908694 podStartE2EDuration="38.315317005s" totalImagesPullingTime="30.247408311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:17.314829715 +0000 UTC m=+38.736548948" watchObservedRunningTime="2026-07-17 21:28:17.315317005 +0000 UTC m=+38.737036238" Jul 17 21:28:17.457809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:17.457778 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:18.017525 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:18.017488 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:18.017706 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:18.017590 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:28:18.053666 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:18.053632 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:18.255102 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:18.255071 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:18.456348 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:18.456256 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:18.784882 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:18.784854 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:18.989284 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:18.989248 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:19.019029 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:19.018993 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:19.019205 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:19.019089 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:28:19.019205 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:19.019181 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:19.019319 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:19.019283 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:28:20.017523 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:20.017490 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:20.017915 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:20.017594 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:28:21.018199 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:21.018162 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:21.018573 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:21.018209 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:21.018573 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:21.018306 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:28:21.018573 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:21.018425 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:28:22.018446 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:22.018413 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:22.018820 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:22.018512 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:28:23.018068 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:23.018032 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:23.018232 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:23.018173 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:28:23.018232 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:23.018213 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:23.018315 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:23.018298 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:28:24.017568 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:24.017526 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:24.017972 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:24.017629 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:28:25.018229 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:25.018197 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:25.018718 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:25.018309 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:28:25.018718 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:25.018376 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:25.018718 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:25.018513 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:28:26.017981 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:26.017933 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:26.018195 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:26.018079 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:28:26.239516 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:26.239478 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:26.370814 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:26.370710 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:26.479995 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:26.479962 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:26.571658 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:26.571604 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:26.705613 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:26.705505 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:26.805593 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:26.805542 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:27.018213 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:27.018180 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:27.018420 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:27.018292 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:28:27.018420 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:27.018349 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:27.018546 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:27.018469 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:28:27.389320 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:27.389231 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:27.489627 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:27.489590 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:27.863814 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:27.863772 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fl9qj"] Jul 17 21:28:27.863986 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:27.863945 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:27.864095 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:27.864067 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:28:27.867135 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:27.867099 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n7c88"] Jul 17 21:28:27.867270 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:27.867247 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:27.867408 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:27.867355 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:28:27.868026 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:27.868001 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d2x4m"] Jul 17 21:28:27.868136 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:27.868098 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:27.868215 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:27.868188 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:28:28.886879 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:28.886839 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:29.018332 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:29.018302 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:29.018520 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:29.018409 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:28:29.062256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:29.062225 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:29.118190 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:29.118155 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:30.018408 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:30.018354 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:30.018408 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:30.018354 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:30.018864 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:30.018477 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fl9qj" podUID="f44bc251-268b-42f5-b9d3-a742151884f9" Jul 17 21:28:30.018864 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:30.018615 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d2x4m" podUID="896ed8fb-96fc-4834-83a6-807a16a340b3" Jul 17 21:28:30.396464 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:30.396428 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:30.506843 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:30.506804 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:31.017824 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:31.017783 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:31.018132 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:31.017921 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n7c88" podUID="c85213e4-c44c-418e-ae68-c74526ae6fa5" Jul 17 21:28:31.580886 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:31.580856 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:31.682482 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:31.682255 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:31.940569 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:31.940485 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-56.ec2.internal" event="NodeReady" Jul 17 21:28:31.940746 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:31.940603 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 17 21:28:31.982330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:31.982292 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bc4774985-2r78g"] Jul 17 21:28:32.004249 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.004212 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kzdwp"] Jul 17 21:28:32.004443 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.004305 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.006989 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.006957 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 17 21:28:32.007133 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.007014 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bd8xd\"" Jul 17 21:28:32.007133 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.007051 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 17 21:28:32.007133 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.007048 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 17 21:28:32.013890 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.013783 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 17 21:28:32.017569 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.017544 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ztjvd"] Jul 17 21:28:32.017690 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.017626 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:32.017733 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.017626 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kzdwp" Jul 17 21:28:32.017854 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.017835 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:32.021175 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.021151 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 17 21:28:32.021421 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.021382 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 21:28:32.021521 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.021510 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dhbkk\"" Jul 17 21:28:32.021607 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.021511 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 17 21:28:32.021713 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.021630 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 21:28:32.021798 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.021784 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5fkx7\"" Jul 17 21:28:32.021841 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.021826 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 17 21:28:32.039701 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.039677 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bc4774985-2r78g"] Jul 17 21:28:32.039814 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.039713 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kzdwp"] Jul 17 21:28:32.039814 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.039734 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ztjvd"] Jul 17 21:28:32.039814 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.039777 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ztjvd" Jul 17 21:28:32.042306 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.042289 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 17 21:28:32.042645 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.042632 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qt4k4\"" Jul 17 21:28:32.042702 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.042634 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 17 21:28:32.108805 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.108764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1cc4c446-5adf-46b0-86ce-f23ca75bb865-installation-pull-secrets\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.108989 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.108837 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1cc4c446-5adf-46b0-86ce-f23ca75bb865-ca-trust-extracted\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.108989 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.108862 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1cc4c446-5adf-46b0-86ce-f23ca75bb865-registry-certificates\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.108989 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.108885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cc4c446-5adf-46b0-86ce-f23ca75bb865-trusted-ca\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.108989 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.108909 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b229eabe-6732-4bdf-b6c8-57555f105efa-tmp-dir\") pod \"dns-default-ztjvd\" (UID: \"b229eabe-6732-4bdf-b6c8-57555f105efa\") " pod="openshift-dns/dns-default-ztjvd" Jul 17 21:28:32.108989 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.108958 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1cc4c446-5adf-46b0-86ce-f23ca75bb865-image-registry-private-configuration\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.108989 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.108980 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b229eabe-6732-4bdf-b6c8-57555f105efa-metrics-tls\") pod \"dns-default-ztjvd\" (UID: \"b229eabe-6732-4bdf-b6c8-57555f105efa\") " pod="openshift-dns/dns-default-ztjvd" Jul 17 21:28:32.109276 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.109004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2d6f2a4-1393-4284-9529-3af4fdf10f71-cert\") pod \"ingress-canary-kzdwp\" (UID: \"e2d6f2a4-1393-4284-9529-3af4fdf10f71\") " pod="openshift-ingress-canary/ingress-canary-kzdwp" Jul 17 21:28:32.109276 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.109055 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h95pm\" (UniqueName: \"kubernetes.io/projected/e2d6f2a4-1393-4284-9529-3af4fdf10f71-kube-api-access-h95pm\") pod \"ingress-canary-kzdwp\" (UID: \"e2d6f2a4-1393-4284-9529-3af4fdf10f71\") " pod="openshift-ingress-canary/ingress-canary-kzdwp" Jul 17 21:28:32.109276 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.109078 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cc4c446-5adf-46b0-86ce-f23ca75bb865-registry-tls\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.109276 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.109101 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl8dh\" (UniqueName: \"kubernetes.io/projected/b229eabe-6732-4bdf-b6c8-57555f105efa-kube-api-access-hl8dh\") pod \"dns-default-ztjvd\" (UID: \"b229eabe-6732-4bdf-b6c8-57555f105efa\") " pod="openshift-dns/dns-default-ztjvd" Jul 17 21:28:32.109276 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.109163 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b229eabe-6732-4bdf-b6c8-57555f105efa-config-volume\") pod \"dns-default-ztjvd\" (UID: \"b229eabe-6732-4bdf-b6c8-57555f105efa\") " pod="openshift-dns/dns-default-ztjvd" Jul 17 21:28:32.109276 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.109208 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cc4c446-5adf-46b0-86ce-f23ca75bb865-bound-sa-token\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.109276 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.109238 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvgj5\" (UniqueName: \"kubernetes.io/projected/1cc4c446-5adf-46b0-86ce-f23ca75bb865-kube-api-access-vvgj5\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.165205 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.165174 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:32.210162 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.210033 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1cc4c446-5adf-46b0-86ce-f23ca75bb865-image-registry-private-configuration\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.210162 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.210075 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b229eabe-6732-4bdf-b6c8-57555f105efa-metrics-tls\") pod \"dns-default-ztjvd\" (UID: \"b229eabe-6732-4bdf-b6c8-57555f105efa\") " pod="openshift-dns/dns-default-ztjvd" Jul 17 21:28:32.210162 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.210095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2d6f2a4-1393-4284-9529-3af4fdf10f71-cert\") pod \"ingress-canary-kzdwp\" (UID: \"e2d6f2a4-1393-4284-9529-3af4fdf10f71\") " pod="openshift-ingress-canary/ingress-canary-kzdwp" Jul 17 21:28:32.210162 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.210147 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h95pm\" (UniqueName: \"kubernetes.io/projected/e2d6f2a4-1393-4284-9529-3af4fdf10f71-kube-api-access-h95pm\") pod \"ingress-canary-kzdwp\" (UID: \"e2d6f2a4-1393-4284-9529-3af4fdf10f71\") " pod="openshift-ingress-canary/ingress-canary-kzdwp" Jul 17 21:28:32.210479 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.210173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cc4c446-5adf-46b0-86ce-f23ca75bb865-registry-tls\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.210479 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.210197 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl8dh\" (UniqueName: \"kubernetes.io/projected/b229eabe-6732-4bdf-b6c8-57555f105efa-kube-api-access-hl8dh\") pod \"dns-default-ztjvd\" (UID: \"b229eabe-6732-4bdf-b6c8-57555f105efa\") " pod="openshift-dns/dns-default-ztjvd" Jul 17 21:28:32.210479 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.210227 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b229eabe-6732-4bdf-b6c8-57555f105efa-config-volume\") pod \"dns-default-ztjvd\" (UID: \"b229eabe-6732-4bdf-b6c8-57555f105efa\") " pod="openshift-dns/dns-default-ztjvd" Jul 17 21:28:32.210479 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.210254 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cc4c446-5adf-46b0-86ce-f23ca75bb865-bound-sa-token\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.210479 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.210278 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvgj5\" (UniqueName: \"kubernetes.io/projected/1cc4c446-5adf-46b0-86ce-f23ca75bb865-kube-api-access-vvgj5\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.210479 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.210326 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1cc4c446-5adf-46b0-86ce-f23ca75bb865-installation-pull-secrets\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.210479 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.210380 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1cc4c446-5adf-46b0-86ce-f23ca75bb865-ca-trust-extracted\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.210479 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.210430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1cc4c446-5adf-46b0-86ce-f23ca75bb865-registry-certificates\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.210479 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.210455 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cc4c446-5adf-46b0-86ce-f23ca75bb865-trusted-ca\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.210479 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.210479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b229eabe-6732-4bdf-b6c8-57555f105efa-tmp-dir\") pod \"dns-default-ztjvd\" (UID: \"b229eabe-6732-4bdf-b6c8-57555f105efa\") " pod="openshift-dns/dns-default-ztjvd" Jul 17 21:28:32.210907 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.210774 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b229eabe-6732-4bdf-b6c8-57555f105efa-tmp-dir\") pod \"dns-default-ztjvd\" (UID: \"b229eabe-6732-4bdf-b6c8-57555f105efa\") " pod="openshift-dns/dns-default-ztjvd" Jul 17 21:28:32.211874 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.211293 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b229eabe-6732-4bdf-b6c8-57555f105efa-config-volume\") pod \"dns-default-ztjvd\" (UID: \"b229eabe-6732-4bdf-b6c8-57555f105efa\") " pod="openshift-dns/dns-default-ztjvd" Jul 17 21:28:32.211874 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.211626 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1cc4c446-5adf-46b0-86ce-f23ca75bb865-registry-certificates\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.211874 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.211690 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1cc4c446-5adf-46b0-86ce-f23ca75bb865-ca-trust-extracted\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.211874 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.211828 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cc4c446-5adf-46b0-86ce-f23ca75bb865-trusted-ca\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.215100 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.215066 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1cc4c446-5adf-46b0-86ce-f23ca75bb865-image-registry-private-configuration\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.215274 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.215254 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1cc4c446-5adf-46b0-86ce-f23ca75bb865-installation-pull-secrets\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.215348 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.215263 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b229eabe-6732-4bdf-b6c8-57555f105efa-metrics-tls\") pod \"dns-default-ztjvd\" (UID: \"b229eabe-6732-4bdf-b6c8-57555f105efa\") " pod="openshift-dns/dns-default-ztjvd" Jul 17 21:28:32.215348 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.215327 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cc4c446-5adf-46b0-86ce-f23ca75bb865-registry-tls\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.215468 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.215369 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2d6f2a4-1393-4284-9529-3af4fdf10f71-cert\") pod \"ingress-canary-kzdwp\" (UID: \"e2d6f2a4-1393-4284-9529-3af4fdf10f71\") " pod="openshift-ingress-canary/ingress-canary-kzdwp" Jul 17 21:28:32.218772 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.218752 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl8dh\" (UniqueName: \"kubernetes.io/projected/b229eabe-6732-4bdf-b6c8-57555f105efa-kube-api-access-hl8dh\") pod \"dns-default-ztjvd\" (UID: \"b229eabe-6732-4bdf-b6c8-57555f105efa\") " pod="openshift-dns/dns-default-ztjvd" Jul 17 21:28:32.219146 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.219129 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h95pm\" (UniqueName: \"kubernetes.io/projected/e2d6f2a4-1393-4284-9529-3af4fdf10f71-kube-api-access-h95pm\") pod \"ingress-canary-kzdwp\" (UID: \"e2d6f2a4-1393-4284-9529-3af4fdf10f71\") " pod="openshift-ingress-canary/ingress-canary-kzdwp" Jul 17 21:28:32.219485 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.219468 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cc4c446-5adf-46b0-86ce-f23ca75bb865-bound-sa-token\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.219927 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.219909 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvgj5\" (UniqueName: \"kubernetes.io/projected/1cc4c446-5adf-46b0-86ce-f23ca75bb865-kube-api-access-vvgj5\") pod \"image-registry-5bc4774985-2r78g\" (UID: \"1cc4c446-5adf-46b0-86ce-f23ca75bb865\") " pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.266115 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.266081 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:32.317245 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.317210 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:32.333036 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.333005 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kzdwp" Jul 17 21:28:32.347586 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.347560 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ztjvd" Jul 17 21:28:32.531458 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.531423 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bc4774985-2r78g"] Jul 17 21:28:32.538647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.538615 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kzdwp"] Jul 17 21:28:32.542319 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:28:32.542293 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d6f2a4_1393_4284_9529_3af4fdf10f71.slice/crio-524d0508d4a1df813716312b020faf8dac9b4dcfb7bd75ff70d1bf3c44c49804 WatchSource:0}: Error finding container 524d0508d4a1df813716312b020faf8dac9b4dcfb7bd75ff70d1bf3c44c49804: Status 404 returned error can't find the container with id 524d0508d4a1df813716312b020faf8dac9b4dcfb7bd75ff70d1bf3c44c49804 Jul 17 21:28:32.554573 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:32.554549 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ztjvd"] Jul 17 21:28:32.557946 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:28:32.557914 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb229eabe_6732_4bdf_b6c8_57555f105efa.slice/crio-2cdc0142e52932d418cdd8c3a6d7721dbc52f6dc21e320555e960f6ae7722971 WatchSource:0}: Error finding container 2cdc0142e52932d418cdd8c3a6d7721dbc52f6dc21e320555e960f6ae7722971: Status 404 returned error can't find the container with id 2cdc0142e52932d418cdd8c3a6d7721dbc52f6dc21e320555e960f6ae7722971 Jul 17 21:28:33.021932 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:33.021642 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:33.025128 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:33.025102 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 21:28:33.026250 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:33.026222 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7k8fl\"" Jul 17 21:28:33.026373 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:33.026230 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 21:28:33.331485 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:33.331362 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ztjvd" event={"ID":"b229eabe-6732-4bdf-b6c8-57555f105efa","Type":"ContainerStarted","Data":"2cdc0142e52932d418cdd8c3a6d7721dbc52f6dc21e320555e960f6ae7722971"} Jul 17 21:28:33.332949 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:33.332878 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kzdwp" event={"ID":"e2d6f2a4-1393-4284-9529-3af4fdf10f71","Type":"ContainerStarted","Data":"524d0508d4a1df813716312b020faf8dac9b4dcfb7bd75ff70d1bf3c44c49804"} Jul 17 21:28:33.334735 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:33.334705 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bc4774985-2r78g" event={"ID":"1cc4c446-5adf-46b0-86ce-f23ca75bb865","Type":"ContainerStarted","Data":"219f3da6cf5674f7e6001a54272208a5be10656cfef95deedd02573eea666ce5"} Jul 17 21:28:33.334849 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:33.334753 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bc4774985-2r78g" event={"ID":"1cc4c446-5adf-46b0-86ce-f23ca75bb865","Type":"ContainerStarted","Data":"4fba88db42438061a8ee30db6c9d74c642b8fb226d1fc8bebaf9f155ff463717"} Jul 17 21:28:33.335425 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:33.335406 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:33.402243 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:33.402207 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:33.454034 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:33.453946 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:34.001549 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:34.001513 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:34.097987 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:34.097944 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:34.440280 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:34.440245 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:34.494600 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:34.494563 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:35.341436 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:35.341327 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ztjvd" event={"ID":"b229eabe-6732-4bdf-b6c8-57555f105efa","Type":"ContainerStarted","Data":"bf1715783fdbfaf96c8baf9b194f07a96bc313cd1069991a479f1d81657ce87f"} Jul 17 21:28:35.341436 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:35.341425 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ztjvd" event={"ID":"b229eabe-6732-4bdf-b6c8-57555f105efa","Type":"ContainerStarted","Data":"c0ac45502612420cf8953d8cb1c3ea9aa7034ba0ebd549fb7dbc5095ac99a673"} Jul 17 21:28:35.341935 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:35.341461 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ztjvd" Jul 17 21:28:35.344632 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:35.344596 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kzdwp" event={"ID":"e2d6f2a4-1393-4284-9529-3af4fdf10f71","Type":"ContainerStarted","Data":"557e514027ad42d2de592b8cb0a12ef0fb368149ffd7714c105ded60ab578e3b"} Jul 17 21:28:35.358833 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:35.358786 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ztjvd" podStartSLOduration=1.879474263 podStartE2EDuration="4.358774051s" totalImagesPullingTime="2.479299788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:35.357806422 +0000 UTC m=+56.779525655" watchObservedRunningTime="2026-07-17 21:28:35.358774051 +0000 UTC m=+56.780493262" Jul 17 21:28:35.358967 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:35.358914 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5bc4774985-2r78g" podStartSLOduration=9.358908426 podStartE2EDuration="9.358908426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:33.359545726 +0000 UTC m=+54.781264987" watchObservedRunningTime="2026-07-17 21:28:35.358908426 +0000 UTC m=+56.780627653" Jul 17 21:28:35.374631 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:35.374459 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kzdwp" podStartSLOduration=1.876796057 podStartE2EDuration="4.374449063s" totalImagesPullingTime="2.497653006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:35.374421811 +0000 UTC m=+56.796141045" watchObservedRunningTime="2026-07-17 21:28:35.374449063 +0000 UTC m=+56.796168279" Jul 17 21:28:36.942307 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:36.942269 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:37.003242 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:37.003215 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:39.019162 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:39.019128 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:39.077976 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:39.077938 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:39.838183 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:39.838084 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:39.961846 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:39.961811 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5d5nk"] Jul 17 21:28:39.964524 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:39.964508 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:39.967020 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:39.966988 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 17 21:28:39.967218 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:39.967094 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 17 21:28:39.967218 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:39.967107 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 17 21:28:39.968068 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:39.968049 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jn2hx\"" Jul 17 21:28:39.968127 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:39.968068 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 17 21:28:39.977893 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:39.977868 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5d5nk"] Jul 17 21:28:39.978129 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:39.978107 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:40.066022 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.065993 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:40.068033 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.068004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad6b74e2-eda2-430c-838a-5a702016c7bd-data-volume\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:40.068033 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.068035 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gq8d\" (UniqueName: \"kubernetes.io/projected/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-api-access-9gq8d\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:40.068241 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.068057 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad6b74e2-eda2-430c-838a-5a702016c7bd-crio-socket\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:40.068241 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.068083 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad6b74e2-eda2-430c-838a-5a702016c7bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:40.068241 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.068185 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:40.138751 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.138668 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:40.169410 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.169355 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad6b74e2-eda2-430c-838a-5a702016c7bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:40.169573 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.169463 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:40.169573 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:40.169508 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 21:28:40.169573 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.169536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad6b74e2-eda2-430c-838a-5a702016c7bd-data-volume\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:40.169692 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:40.169577 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 21:28:40.169692 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:40.169579 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad6b74e2-eda2-430c-838a-5a702016c7bd-insights-runtime-extractor-tls podName:ad6b74e2-eda2-430c-838a-5a702016c7bd nodeName:}" failed. No retries permitted until 2026-07-17 21:28:40.669559212 +0000 UTC m=+62.091278437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad6b74e2-eda2-430c-838a-5a702016c7bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5d5nk" (UID: "ad6b74e2-eda2-430c-838a-5a702016c7bd") : secret "insights-runtime-extractor-tls" not found Jul 17 21:28:40.169692 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.169640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gq8d\" (UniqueName: \"kubernetes.io/projected/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-api-access-9gq8d\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:40.169692 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.169680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad6b74e2-eda2-430c-838a-5a702016c7bd-crio-socket\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:40.169838 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:40.169711 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-rbac-proxy-cm podName:ad6b74e2-eda2-430c-838a-5a702016c7bd nodeName:}" failed. No retries permitted until 2026-07-17 21:28:40.669687672 +0000 UTC m=+62.091406886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5d5nk" (UID: "ad6b74e2-eda2-430c-838a-5a702016c7bd") : configmap "kube-rbac-proxy" not found Jul 17 21:28:40.169838 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.169737 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad6b74e2-eda2-430c-838a-5a702016c7bd-crio-socket\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:40.169908 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.169844 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad6b74e2-eda2-430c-838a-5a702016c7bd-data-volume\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:40.187607 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.187584 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gq8d\" (UniqueName: \"kubernetes.io/projected/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-api-access-9gq8d\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:40.280772 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.280743 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rhkgr" Jul 17 21:28:40.435134 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.435033 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:40.501625 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.501591 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:40.675214 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.675181 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:40.675415 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:40.675257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad6b74e2-eda2-430c-838a-5a702016c7bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:40.675415 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:40.675315 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 21:28:40.675415 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:40.675365 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 21:28:40.675415 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:40.675382 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-rbac-proxy-cm podName:ad6b74e2-eda2-430c-838a-5a702016c7bd nodeName:}" failed. No retries permitted until 2026-07-17 21:28:41.675364317 +0000 UTC m=+63.097083529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5d5nk" (UID: "ad6b74e2-eda2-430c-838a-5a702016c7bd") : configmap "kube-rbac-proxy" not found Jul 17 21:28:40.675557 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:40.675435 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad6b74e2-eda2-430c-838a-5a702016c7bd-insights-runtime-extractor-tls podName:ad6b74e2-eda2-430c-838a-5a702016c7bd nodeName:}" failed. No retries permitted until 2026-07-17 21:28:41.675421204 +0000 UTC m=+63.097140416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad6b74e2-eda2-430c-838a-5a702016c7bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5d5nk" (UID: "ad6b74e2-eda2-430c-838a-5a702016c7bd") : secret "insights-runtime-extractor-tls" not found Jul 17 21:28:41.681310 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:41.681243 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad6b74e2-eda2-430c-838a-5a702016c7bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:41.681716 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:41.681342 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:41.681716 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:41.681469 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 21:28:41.681716 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:41.681541 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-rbac-proxy-cm podName:ad6b74e2-eda2-430c-838a-5a702016c7bd nodeName:}" failed. No retries permitted until 2026-07-17 21:28:43.681524431 +0000 UTC m=+65.103243642 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5d5nk" (UID: "ad6b74e2-eda2-430c-838a-5a702016c7bd") : configmap "kube-rbac-proxy" not found Jul 17 21:28:41.681716 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:41.681471 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 21:28:41.681716 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:41.681644 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad6b74e2-eda2-430c-838a-5a702016c7bd-insights-runtime-extractor-tls podName:ad6b74e2-eda2-430c-838a-5a702016c7bd nodeName:}" failed. No retries permitted until 2026-07-17 21:28:43.681624554 +0000 UTC m=+65.103343768 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad6b74e2-eda2-430c-838a-5a702016c7bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5d5nk" (UID: "ad6b74e2-eda2-430c-838a-5a702016c7bd") : secret "insights-runtime-extractor-tls" not found Jul 17 21:28:42.093920 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:42.093876 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:42.170165 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:42.170133 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:42.253930 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:42.253897 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:43.699820 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:43.699782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad6b74e2-eda2-430c-838a-5a702016c7bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:43.700223 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:43.699836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:43.700223 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:43.699920 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 21:28:43.700223 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:43.699942 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 21:28:43.700223 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:43.699976 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-rbac-proxy-cm podName:ad6b74e2-eda2-430c-838a-5a702016c7bd nodeName:}" failed. No retries permitted until 2026-07-17 21:28:47.699961386 +0000 UTC m=+69.121680596 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5d5nk" (UID: "ad6b74e2-eda2-430c-838a-5a702016c7bd") : configmap "kube-rbac-proxy" not found Jul 17 21:28:43.700223 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:43.699996 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad6b74e2-eda2-430c-838a-5a702016c7bd-insights-runtime-extractor-tls podName:ad6b74e2-eda2-430c-838a-5a702016c7bd nodeName:}" failed. No retries permitted until 2026-07-17 21:28:47.699983059 +0000 UTC m=+69.121702270 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad6b74e2-eda2-430c-838a-5a702016c7bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5d5nk" (UID: "ad6b74e2-eda2-430c-838a-5a702016c7bd") : secret "insights-runtime-extractor-tls" not found Jul 17 21:28:43.750832 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:43.750808 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ztjvd_b229eabe-6732-4bdf-b6c8-57555f105efa/dns/0.log" Jul 17 21:28:43.767466 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:43.767430 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:43.864739 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:43.864702 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:43.928131 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:43.928097 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:43.950351 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:43.950274 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ztjvd_b229eabe-6732-4bdf-b6c8-57555f105efa/kube-rbac-proxy/0.log" Jul 17 21:28:44.065274 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.065240 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:44.150211 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.150188 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4fgf5_3e6f72d2-a524-4587-a7f5-278128245a09/dns-node-resolver/0.log" Jul 17 21:28:44.481411 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.481367 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:44.557476 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.557445 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:44.655930 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.655898 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:44.707205 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.707168 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret\") pod \"global-pull-secret-syncer-fl9qj\" (UID: \"f44bc251-268b-42f5-b9d3-a742151884f9\") " pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:44.707585 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.707224 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs\") pod \"network-metrics-daemon-d2x4m\" (UID: \"896ed8fb-96fc-4834-83a6-807a16a340b3\") " pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:44.709782 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.709766 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 21:28:44.709848 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.709766 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 21:28:44.712056 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.712035 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:44.720351 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.720327 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f44bc251-268b-42f5-b9d3-a742151884f9-original-pull-secret\") pod \"global-pull-secret-syncer-fl9qj\" (UID: \"f44bc251-268b-42f5-b9d3-a742151884f9\") " pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:44.720478 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.720449 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896ed8fb-96fc-4834-83a6-807a16a340b3-metrics-certs\") pod \"network-metrics-daemon-d2x4m\" (UID: \"896ed8fb-96fc-4834-83a6-807a16a340b3\") " pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:44.750270 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.750204 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5bc4774985-2r78g_1cc4c446-5adf-46b0-86ce-f23ca75bb865/registry/0.log" Jul 17 21:28:44.807569 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.807538 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtgm2\" (UniqueName: \"kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2\") pod \"network-check-target-n7c88\" (UID: \"c85213e4-c44c-418e-ae68-c74526ae6fa5\") " pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:44.810619 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.810589 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 21:28:44.820859 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.820838 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 21:28:44.831568 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.831539 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtgm2\" (UniqueName: \"kubernetes.io/projected/c85213e4-c44c-418e-ae68-c74526ae6fa5-kube-api-access-mtgm2\") pod \"network-check-target-n7c88\" (UID: \"c85213e4-c44c-418e-ae68-c74526ae6fa5\") " pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:44.927331 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.927294 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fl9qj" Jul 17 21:28:44.940137 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.940114 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5fkx7\"" Jul 17 21:28:44.948275 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:44.948250 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d2x4m" Jul 17 21:28:45.035775 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:45.035541 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7k8fl\"" Jul 17 21:28:45.043232 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:45.042951 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:45.108226 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:45.108194 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fl9qj"] Jul 17 21:28:45.111971 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:28:45.111940 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44bc251_268b_42f5_b9d3_a742151884f9.slice/crio-e8124243ca2236a2c25a319e7f5decb08a3cfeb9844d6608920fc67ee23ad541 WatchSource:0}: Error finding container e8124243ca2236a2c25a319e7f5decb08a3cfeb9844d6608920fc67ee23ad541: Status 404 returned error can't find the container with id e8124243ca2236a2c25a319e7f5decb08a3cfeb9844d6608920fc67ee23ad541 Jul 17 21:28:45.124733 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:45.124702 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d2x4m"] Jul 17 21:28:45.129646 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:28:45.129595 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896ed8fb_96fc_4834_83a6_807a16a340b3.slice/crio-78d62ae87e31e16de1e55576bb5240dbc40a9b84d87ab62b46eb7cfa37484542 WatchSource:0}: Error finding container 78d62ae87e31e16de1e55576bb5240dbc40a9b84d87ab62b46eb7cfa37484542: Status 404 returned error can't find the container with id 78d62ae87e31e16de1e55576bb5240dbc40a9b84d87ab62b46eb7cfa37484542 Jul 17 21:28:45.201822 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:45.201799 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n7c88"] Jul 17 21:28:45.203965 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:28:45.203938 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85213e4_c44c_418e_ae68_c74526ae6fa5.slice/crio-b5abb1bda73d2190498d4d356c6041bad9b87e0cda38951cae134b5277741886 WatchSource:0}: Error finding container b5abb1bda73d2190498d4d356c6041bad9b87e0cda38951cae134b5277741886: Status 404 returned error can't find the container with id b5abb1bda73d2190498d4d356c6041bad9b87e0cda38951cae134b5277741886 Jul 17 21:28:45.349614 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:45.349522 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6mqj2_fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694/node-ca/0.log" Jul 17 21:28:45.349614 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:45.349608 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ztjvd" Jul 17 21:28:45.369991 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:45.369951 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d2x4m" event={"ID":"896ed8fb-96fc-4834-83a6-807a16a340b3","Type":"ContainerStarted","Data":"78d62ae87e31e16de1e55576bb5240dbc40a9b84d87ab62b46eb7cfa37484542"} Jul 17 21:28:45.371124 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:45.371094 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fl9qj" event={"ID":"f44bc251-268b-42f5-b9d3-a742151884f9","Type":"ContainerStarted","Data":"e8124243ca2236a2c25a319e7f5decb08a3cfeb9844d6608920fc67ee23ad541"} Jul 17 21:28:45.372153 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:45.372136 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n7c88" event={"ID":"c85213e4-c44c-418e-ae68-c74526ae6fa5","Type":"ContainerStarted","Data":"b5abb1bda73d2190498d4d356c6041bad9b87e0cda38951cae134b5277741886"} Jul 17 21:28:46.070944 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.070860 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:46.124726 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.124675 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:46.149229 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.149173 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kzdwp_e2d6f2a4-1393-4284-9529-3af4fdf10f71/serve-healthcheck-canary/0.log" Jul 17 21:28:46.238995 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.238964 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:46.448089 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.448009 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:46.489334 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.489269 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k"] Jul 17 21:28:46.492917 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.492899 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" Jul 17 21:28:46.496259 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.496221 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 17 21:28:46.496777 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.496757 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 17 21:28:46.497375 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.497356 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 17 21:28:46.497491 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.497465 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 17 21:28:46.497609 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.497594 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hntws\"" Jul 17 21:28:46.497907 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.497759 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 17 21:28:46.511189 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.511149 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k"] Jul 17 21:28:46.543433 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.543380 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zz8sh"] Jul 17 21:28:46.545527 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.545502 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.548442 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.548422 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 17 21:28:46.548567 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.548503 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cbqw8\"" Jul 17 21:28:46.548919 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.548885 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 17 21:28:46.549173 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.549159 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 17 21:28:46.552860 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.552837 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:46.622101 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.622013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/31b165ae-1d35-4dfa-8ff9-f693f65bc628-node-exporter-tls\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.622101 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.622068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/31b165ae-1d35-4dfa-8ff9-f693f65bc628-node-exporter-accelerators-collector-config\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.622311 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.622161 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31b165ae-1d35-4dfa-8ff9-f693f65bc628-sys\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.622311 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.622213 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr6ph\" (UniqueName: \"kubernetes.io/projected/513ad778-cfa6-4e4f-8a6f-e8c0a08492f4-kube-api-access-mr6ph\") pod \"openshift-state-metrics-79f6958cdf-w8z5k\" (UID: \"513ad778-cfa6-4e4f-8a6f-e8c0a08492f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" Jul 17 21:28:46.622311 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.622247 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/513ad778-cfa6-4e4f-8a6f-e8c0a08492f4-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-w8z5k\" (UID: \"513ad778-cfa6-4e4f-8a6f-e8c0a08492f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" Jul 17 21:28:46.622311 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.622281 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/31b165ae-1d35-4dfa-8ff9-f693f65bc628-node-exporter-textfile\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.622311 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.622312 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/513ad778-cfa6-4e4f-8a6f-e8c0a08492f4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-w8z5k\" (UID: \"513ad778-cfa6-4e4f-8a6f-e8c0a08492f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" Jul 17 21:28:46.622568 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.622347 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgjkn\" (UniqueName: \"kubernetes.io/projected/31b165ae-1d35-4dfa-8ff9-f693f65bc628-kube-api-access-jgjkn\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.622568 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.622367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/31b165ae-1d35-4dfa-8ff9-f693f65bc628-root\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.622568 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.622407 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31b165ae-1d35-4dfa-8ff9-f693f65bc628-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.622568 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.622466 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/513ad778-cfa6-4e4f-8a6f-e8c0a08492f4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-w8z5k\" (UID: \"513ad778-cfa6-4e4f-8a6f-e8c0a08492f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" Jul 17 21:28:46.622568 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.622526 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/31b165ae-1d35-4dfa-8ff9-f693f65bc628-node-exporter-wtmp\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.622749 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.622599 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31b165ae-1d35-4dfa-8ff9-f693f65bc628-metrics-client-ca\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.639358 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.639334 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:46.723651 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.723613 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31b165ae-1d35-4dfa-8ff9-f693f65bc628-metrics-client-ca\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.723838 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.723663 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/31b165ae-1d35-4dfa-8ff9-f693f65bc628-node-exporter-tls\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.723838 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.723703 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/31b165ae-1d35-4dfa-8ff9-f693f65bc628-node-exporter-accelerators-collector-config\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.723838 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.723734 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31b165ae-1d35-4dfa-8ff9-f693f65bc628-sys\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.723838 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.723761 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr6ph\" (UniqueName: \"kubernetes.io/projected/513ad778-cfa6-4e4f-8a6f-e8c0a08492f4-kube-api-access-mr6ph\") pod \"openshift-state-metrics-79f6958cdf-w8z5k\" (UID: \"513ad778-cfa6-4e4f-8a6f-e8c0a08492f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" Jul 17 21:28:46.723838 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.723790 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/513ad778-cfa6-4e4f-8a6f-e8c0a08492f4-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-w8z5k\" (UID: \"513ad778-cfa6-4e4f-8a6f-e8c0a08492f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" Jul 17 21:28:46.723838 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.723819 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/31b165ae-1d35-4dfa-8ff9-f693f65bc628-node-exporter-textfile\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.724138 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.723847 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/513ad778-cfa6-4e4f-8a6f-e8c0a08492f4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-w8z5k\" (UID: \"513ad778-cfa6-4e4f-8a6f-e8c0a08492f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" Jul 17 21:28:46.724138 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.723893 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgjkn\" (UniqueName: \"kubernetes.io/projected/31b165ae-1d35-4dfa-8ff9-f693f65bc628-kube-api-access-jgjkn\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.724138 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.723922 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/31b165ae-1d35-4dfa-8ff9-f693f65bc628-root\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.724138 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.723948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31b165ae-1d35-4dfa-8ff9-f693f65bc628-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.724138 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.723974 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/513ad778-cfa6-4e4f-8a6f-e8c0a08492f4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-w8z5k\" (UID: \"513ad778-cfa6-4e4f-8a6f-e8c0a08492f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" Jul 17 21:28:46.724138 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.724006 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/31b165ae-1d35-4dfa-8ff9-f693f65bc628-node-exporter-wtmp\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.724452 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.724216 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/31b165ae-1d35-4dfa-8ff9-f693f65bc628-node-exporter-wtmp\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.724856 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.724831 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31b165ae-1d35-4dfa-8ff9-f693f65bc628-metrics-client-ca\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.725902 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.725276 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/31b165ae-1d35-4dfa-8ff9-f693f65bc628-root\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.725902 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.725466 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/31b165ae-1d35-4dfa-8ff9-f693f65bc628-node-exporter-textfile\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.726058 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.725946 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/31b165ae-1d35-4dfa-8ff9-f693f65bc628-node-exporter-accelerators-collector-config\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.726231 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.726206 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31b165ae-1d35-4dfa-8ff9-f693f65bc628-sys\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.726341 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:46.726321 2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 17 21:28:46.726458 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:46.726410 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/513ad778-cfa6-4e4f-8a6f-e8c0a08492f4-openshift-state-metrics-tls podName:513ad778-cfa6-4e4f-8a6f-e8c0a08492f4 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:47.226370333 +0000 UTC m=+68.648089544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/513ad778-cfa6-4e4f-8a6f-e8c0a08492f4-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-w8z5k" (UID: "513ad778-cfa6-4e4f-8a6f-e8c0a08492f4") : secret "openshift-state-metrics-tls" not found Jul 17 21:28:46.726875 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.726793 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/513ad778-cfa6-4e4f-8a6f-e8c0a08492f4-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-w8z5k\" (UID: \"513ad778-cfa6-4e4f-8a6f-e8c0a08492f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" Jul 17 21:28:46.728599 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.728566 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31b165ae-1d35-4dfa-8ff9-f693f65bc628-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.728696 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.728654 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/513ad778-cfa6-4e4f-8a6f-e8c0a08492f4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-w8z5k\" (UID: \"513ad778-cfa6-4e4f-8a6f-e8c0a08492f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" Jul 17 21:28:46.728999 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.728973 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/31b165ae-1d35-4dfa-8ff9-f693f65bc628-node-exporter-tls\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.740798 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.739240 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr6ph\" (UniqueName: \"kubernetes.io/projected/513ad778-cfa6-4e4f-8a6f-e8c0a08492f4-kube-api-access-mr6ph\") pod \"openshift-state-metrics-79f6958cdf-w8z5k\" (UID: \"513ad778-cfa6-4e4f-8a6f-e8c0a08492f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" Jul 17 21:28:46.740798 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.739384 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgjkn\" (UniqueName: \"kubernetes.io/projected/31b165ae-1d35-4dfa-8ff9-f693f65bc628-kube-api-access-jgjkn\") pod \"node-exporter-zz8sh\" (UID: \"31b165ae-1d35-4dfa-8ff9-f693f65bc628\") " pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.741100 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.741080 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:46.845525 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.845488 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:46.857459 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.857427 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zz8sh" Jul 17 21:28:46.870503 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:28:46.870465 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b165ae_1d35_4dfa_8ff9_f693f65bc628.slice/crio-5ae9daac9e814d145f9861e95f5bf6da1c21d2b0d29fc09bce6cf65b5fce0107 WatchSource:0}: Error finding container 5ae9daac9e814d145f9861e95f5bf6da1c21d2b0d29fc09bce6cf65b5fce0107: Status 404 returned error can't find the container with id 5ae9daac9e814d145f9861e95f5bf6da1c21d2b0d29fc09bce6cf65b5fce0107 Jul 17 21:28:46.996433 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:46.996378 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:47.097459 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.097418 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:47.228167 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.228082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/513ad778-cfa6-4e4f-8a6f-e8c0a08492f4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-w8z5k\" (UID: \"513ad778-cfa6-4e4f-8a6f-e8c0a08492f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" Jul 17 21:28:47.230840 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.230816 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/513ad778-cfa6-4e4f-8a6f-e8c0a08492f4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-w8z5k\" (UID: \"513ad778-cfa6-4e4f-8a6f-e8c0a08492f4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" Jul 17 21:28:47.253924 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.253827 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:47.355800 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.355769 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:47.382989 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.382950 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d2x4m" event={"ID":"896ed8fb-96fc-4834-83a6-807a16a340b3","Type":"ContainerStarted","Data":"9312d9adb429729fdb63a28134ce1938389a4f06cda96b3f1fc66f5da90dff12"} Jul 17 21:28:47.383168 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.383017 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d2x4m" event={"ID":"896ed8fb-96fc-4834-83a6-807a16a340b3","Type":"ContainerStarted","Data":"bf344cd45cfd67af22eb791b3407d330b667e3bca65c10a7cdc8a309364b5beb"} Jul 17 21:28:47.385315 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.385276 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zz8sh" event={"ID":"31b165ae-1d35-4dfa-8ff9-f693f65bc628","Type":"ContainerStarted","Data":"5ae9daac9e814d145f9861e95f5bf6da1c21d2b0d29fc09bce6cf65b5fce0107"} Jul 17 21:28:47.404352 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.403492 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-d2x4m" podStartSLOduration=67.242933089 podStartE2EDuration="1m8.403467956s" totalImagesPullingTime="1.160534867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:47.401793394 +0000 UTC m=+68.823512645" watchObservedRunningTime="2026-07-17 21:28:47.403467956 +0000 UTC m=+68.825187189" Jul 17 21:28:47.404352 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.404134 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" Jul 17 21:28:47.533453 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.533412 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:47.583405 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.583346 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k"] Jul 17 21:28:47.613114 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.613080 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:47.617547 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.617519 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 21:28:47.620469 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.620451 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.625424 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.623041 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 17 21:28:47.625424 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.623496 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 17 21:28:47.625424 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.623518 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 17 21:28:47.625424 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.623622 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 17 21:28:47.625424 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.623691 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 17 21:28:47.625424 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.623720 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 17 21:28:47.625424 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.623791 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 17 21:28:47.625424 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.624420 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 17 21:28:47.625424 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.624708 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fvkxq\"" Jul 17 21:28:47.625424 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.624929 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 17 21:28:47.641915 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.641883 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 21:28:47.732539 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.732506 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-config-out\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.732707 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.732555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.732707 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.732583 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.732707 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.732659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.732707 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.732705 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.732935 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.732740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.732935 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.732766 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-web-config\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.732935 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.732795 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:47.732935 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.732815 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.732935 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.732841 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.732935 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.732891 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.732935 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.732916 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad6b74e2-eda2-430c-838a-5a702016c7bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:47.732935 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.732933 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.733310 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.732953 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.733310 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.732968 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pzbv\" (UniqueName: \"kubernetes.io/projected/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-kube-api-access-7pzbv\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.733310 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:47.733095 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 21:28:47.733310 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:47.733154 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-rbac-proxy-cm podName:ad6b74e2-eda2-430c-838a-5a702016c7bd nodeName:}" failed. No retries permitted until 2026-07-17 21:28:55.733139268 +0000 UTC m=+77.154858478 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5d5nk" (UID: "ad6b74e2-eda2-430c-838a-5a702016c7bd") : configmap "kube-rbac-proxy" not found Jul 17 21:28:47.733609 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:47.733588 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 21:28:47.733690 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:47.733654 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad6b74e2-eda2-430c-838a-5a702016c7bd-insights-runtime-extractor-tls podName:ad6b74e2-eda2-430c-838a-5a702016c7bd nodeName:}" failed. No retries permitted until 2026-07-17 21:28:55.733635705 +0000 UTC m=+77.155354916 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad6b74e2-eda2-430c-838a-5a702016c7bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5d5nk" (UID: "ad6b74e2-eda2-430c-838a-5a702016c7bd") : secret "insights-runtime-extractor-tls" not found Jul 17 21:28:47.735630 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.735561 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:47.834362 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.834325 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.834362 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.834363 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.834609 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.834385 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-web-config\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.834609 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.834452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.834609 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.834479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.834609 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.834508 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.834609 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.834536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.834609 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.834556 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.834609 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.834570 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pzbv\" (UniqueName: \"kubernetes.io/projected/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-kube-api-access-7pzbv\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.834609 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.834588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-config-out\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.834609 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.834606 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.835039 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.834624 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.835039 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.834664 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.835520 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.835240 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.835632 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:47.835523 2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 17 21:28:47.835632 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:28:47.835612 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-secret-alertmanager-main-tls podName:aa077b26-a698-4b72-8f63-b8d3edf4d8ee nodeName:}" failed. No retries permitted until 2026-07-17 21:28:48.335588599 +0000 UTC m=+69.757307816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "aa077b26-a698-4b72-8f63-b8d3edf4d8ee") : secret "alertmanager-main-tls" not found Jul 17 21:28:47.842894 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.836796 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.842894 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.839954 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.842894 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.840520 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.842894 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.841335 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.842894 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.841374 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.842894 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.841811 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.842894 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.842686 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.842894 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.842714 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-config-out\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.843234 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.843147 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-web-config\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.846162 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.846136 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:47.853188 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:47.853160 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pzbv\" (UniqueName: \"kubernetes.io/projected/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-kube-api-access-7pzbv\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:48.135409 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:48.135361 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:48.234609 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:48.234571 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:48.334115 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:48.334082 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:48.339377 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:48.339340 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:48.342338 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:48.342314 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aa077b26-a698-4b72-8f63-b8d3edf4d8ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aa077b26-a698-4b72-8f63-b8d3edf4d8ee\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:48.392208 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:48.392115 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" event={"ID":"513ad778-cfa6-4e4f-8a6f-e8c0a08492f4","Type":"ContainerStarted","Data":"5138c7439f323ade7a23c6977dc21878b8746fe14b8acd5b1d81b3d80976b9ec"} Jul 17 21:28:48.392208 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:48.392190 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" event={"ID":"513ad778-cfa6-4e4f-8a6f-e8c0a08492f4","Type":"ContainerStarted","Data":"f082c3767c6293d10e4051254c6fd0c9e40cf3662b6ff87f6ec610388a45fab8"} Jul 17 21:28:48.392462 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:48.392213 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" event={"ID":"513ad778-cfa6-4e4f-8a6f-e8c0a08492f4","Type":"ContainerStarted","Data":"23411fd5147e299d87948dc83bba55a50fe2a5a67ae6fb8b488e8f75e33a13a8"} Jul 17 21:28:48.549794 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:48.549762 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:28:49.436557 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.436526 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:49.498964 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.498921 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:49.564491 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.564441 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:49.591008 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.590971 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-76bb7cbc89-5flq2"] Jul 17 21:28:49.597756 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.597726 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.600518 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.600493 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 17 21:28:49.600672 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.600545 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 17 21:28:49.600672 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.600617 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 17 21:28:49.600790 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.600733 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 17 21:28:49.600847 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.600816 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3572ansp7l1ep\"" Jul 17 21:28:49.600950 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.600936 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-trzb7\"" Jul 17 21:28:49.601333 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.601316 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 17 21:28:49.605018 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.604762 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76bb7cbc89-5flq2"] Jul 17 21:28:49.754441 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.754331 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8785431a-f6f7-4fc6-a97f-268e35ad79e9-secret-grpc-tls\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.754441 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.754410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8785431a-f6f7-4fc6-a97f-268e35ad79e9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.754738 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.754454 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz2nf\" (UniqueName: \"kubernetes.io/projected/8785431a-f6f7-4fc6-a97f-268e35ad79e9-kube-api-access-tz2nf\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.754738 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.754558 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8785431a-f6f7-4fc6-a97f-268e35ad79e9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.754738 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.754583 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8785431a-f6f7-4fc6-a97f-268e35ad79e9-metrics-client-ca\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.754738 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.754620 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8785431a-f6f7-4fc6-a97f-268e35ad79e9-secret-thanos-querier-tls\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.754738 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.754713 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8785431a-f6f7-4fc6-a97f-268e35ad79e9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.754951 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.754789 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8785431a-f6f7-4fc6-a97f-268e35ad79e9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.856056 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.856011 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8785431a-f6f7-4fc6-a97f-268e35ad79e9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.856229 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.856099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8785431a-f6f7-4fc6-a97f-268e35ad79e9-secret-grpc-tls\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.856229 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.856134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8785431a-f6f7-4fc6-a97f-268e35ad79e9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.856229 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.856161 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz2nf\" (UniqueName: \"kubernetes.io/projected/8785431a-f6f7-4fc6-a97f-268e35ad79e9-kube-api-access-tz2nf\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.856368 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.856230 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8785431a-f6f7-4fc6-a97f-268e35ad79e9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.856368 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.856259 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8785431a-f6f7-4fc6-a97f-268e35ad79e9-metrics-client-ca\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.856368 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.856296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8785431a-f6f7-4fc6-a97f-268e35ad79e9-secret-thanos-querier-tls\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.856368 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.856335 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8785431a-f6f7-4fc6-a97f-268e35ad79e9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.857247 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.857215 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8785431a-f6f7-4fc6-a97f-268e35ad79e9-metrics-client-ca\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.859461 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.859412 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8785431a-f6f7-4fc6-a97f-268e35ad79e9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.859666 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.859633 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8785431a-f6f7-4fc6-a97f-268e35ad79e9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.859813 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.859788 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8785431a-f6f7-4fc6-a97f-268e35ad79e9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.860030 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.860008 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8785431a-f6f7-4fc6-a97f-268e35ad79e9-secret-thanos-querier-tls\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.861025 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.861002 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8785431a-f6f7-4fc6-a97f-268e35ad79e9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.861131 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.861011 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8785431a-f6f7-4fc6-a97f-268e35ad79e9-secret-grpc-tls\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.871959 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.871929 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz2nf\" (UniqueName: \"kubernetes.io/projected/8785431a-f6f7-4fc6-a97f-268e35ad79e9-kube-api-access-tz2nf\") pod \"thanos-querier-76bb7cbc89-5flq2\" (UID: \"8785431a-f6f7-4fc6-a97f-268e35ad79e9\") " pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:49.909641 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:49.909604 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:50.012695 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:50.012663 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:50.704542 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:50.704446 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 21:28:50.710216 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:28:50.710181 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa077b26_a698_4b72_8f63_b8d3edf4d8ee.slice/crio-d30c11148f67ef5d434e6675adfc37c7e95f5f52b8918374388667b14f4c66b4 WatchSource:0}: Error finding container d30c11148f67ef5d434e6675adfc37c7e95f5f52b8918374388667b14f4c66b4: Status 404 returned error can't find the container with id d30c11148f67ef5d434e6675adfc37c7e95f5f52b8918374388667b14f4c66b4 Jul 17 21:28:50.721549 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:50.721525 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76bb7cbc89-5flq2"] Jul 17 21:28:50.724302 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:28:50.724239 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8785431a_f6f7_4fc6_a97f_268e35ad79e9.slice/crio-3614814dfd541e4e483c3ea05fc8f4555c1eb96baa5e48394bf26705255b7449 WatchSource:0}: Error finding container 3614814dfd541e4e483c3ea05fc8f4555c1eb96baa5e48394bf26705255b7449: Status 404 returned error can't find the container with id 3614814dfd541e4e483c3ea05fc8f4555c1eb96baa5e48394bf26705255b7449 Jul 17 21:28:51.279204 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.279174 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:51.345580 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.345545 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:51.405803 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.405683 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fl9qj" event={"ID":"f44bc251-268b-42f5-b9d3-a742151884f9","Type":"ContainerStarted","Data":"f6e692daf5b7d7de0085b09d74f2f952769d95b28c9435403853e992d773ad3c"} Jul 17 21:28:51.407902 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.407874 2580 generic.go:356] "Generic (PLEG): container finished" podID="31b165ae-1d35-4dfa-8ff9-f693f65bc628" containerID="51780c07a90d6ed5000f0899bba8dcd8fbaacccbfd7cbd7ab002de1a2b0194f0" exitCode=0 Jul 17 21:28:51.408051 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.407963 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zz8sh" event={"ID":"31b165ae-1d35-4dfa-8ff9-f693f65bc628","Type":"ContainerDied","Data":"51780c07a90d6ed5000f0899bba8dcd8fbaacccbfd7cbd7ab002de1a2b0194f0"} Jul 17 21:28:51.409439 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.409407 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" event={"ID":"8785431a-f6f7-4fc6-a97f-268e35ad79e9","Type":"ContainerStarted","Data":"3614814dfd541e4e483c3ea05fc8f4555c1eb96baa5e48394bf26705255b7449"} Jul 17 21:28:51.411859 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.411830 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n7c88" event={"ID":"c85213e4-c44c-418e-ae68-c74526ae6fa5","Type":"ContainerStarted","Data":"be68f6fe85876fc632c99c5f5d256c4227bfcde9c43b9355011e0d32db4de906"} Jul 17 21:28:51.412032 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.412013 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:28:51.415051 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.415025 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" event={"ID":"513ad778-cfa6-4e4f-8a6f-e8c0a08492f4","Type":"ContainerStarted","Data":"977fe6d672792a06ef84b497af3236e2293db23504a375e1b448362ebc3b5fa7"} Jul 17 21:28:51.416513 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.416489 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa077b26-a698-4b72-8f63-b8d3edf4d8ee","Type":"ContainerStarted","Data":"d30c11148f67ef5d434e6675adfc37c7e95f5f52b8918374388667b14f4c66b4"} Jul 17 21:28:51.422518 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.422479 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fl9qj" podStartSLOduration=67.002540802 podStartE2EDuration="1m12.422466429s" totalImagesPullingTime="5.419925627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:51.421062363 +0000 UTC m=+72.842781597" watchObservedRunningTime="2026-07-17 21:28:51.422466429 +0000 UTC m=+72.844185661" Jul 17 21:28:51.440344 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.440303 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-w8z5k" podStartSLOduration=2.077922285 podStartE2EDuration="5.440286602s" totalImagesPullingTime="3.362364317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:51.439072946 +0000 UTC m=+72.860792185" watchObservedRunningTime="2026-07-17 21:28:51.440286602 +0000 UTC m=+72.862005835" Jul 17 21:28:51.453852 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.453799 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-n7c88" podStartSLOduration=67.13099518 podStartE2EDuration="1m12.45378259s" totalImagesPullingTime="5.32278741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:51.452998986 +0000 UTC m=+72.874718246" watchObservedRunningTime="2026-07-17 21:28:51.45378259 +0000 UTC m=+72.875501822" Jul 17 21:28:51.461875 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.461833 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:51.534692 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.534662 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:51.746425 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.746337 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6798878cfc-9kgql"] Jul 17 21:28:51.750128 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.750103 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.753439 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.753411 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 17 21:28:51.754941 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.754921 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 17 21:28:51.755185 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.755170 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8nqlx\"" Jul 17 21:28:51.755286 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.755190 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 17 21:28:51.755939 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.755618 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 17 21:28:51.756069 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.756052 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 17 21:28:51.761063 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.761038 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 17 21:28:51.764103 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.764079 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6798878cfc-9kgql"] Jul 17 21:28:51.798221 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.798187 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:51.859228 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.859191 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:51.873583 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.873544 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73c52f87-4e3e-4427-8112-f1ca2c2fba25-serving-certs-ca-bundle\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.873756 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.873642 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x78mn\" (UniqueName: \"kubernetes.io/projected/73c52f87-4e3e-4427-8112-f1ca2c2fba25-kube-api-access-x78mn\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.873756 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.873691 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/73c52f87-4e3e-4427-8112-f1ca2c2fba25-telemeter-client-tls\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.873756 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.873722 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73c52f87-4e3e-4427-8112-f1ca2c2fba25-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.873756 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.873747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73c52f87-4e3e-4427-8112-f1ca2c2fba25-metrics-client-ca\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.873987 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.873787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/73c52f87-4e3e-4427-8112-f1ca2c2fba25-federate-client-tls\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.873987 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.873817 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/73c52f87-4e3e-4427-8112-f1ca2c2fba25-secret-telemeter-client\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.873987 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.873850 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73c52f87-4e3e-4427-8112-f1ca2c2fba25-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.961557 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.961522 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:51.974832 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.974797 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x78mn\" (UniqueName: \"kubernetes.io/projected/73c52f87-4e3e-4427-8112-f1ca2c2fba25-kube-api-access-x78mn\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.974986 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.974856 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/73c52f87-4e3e-4427-8112-f1ca2c2fba25-telemeter-client-tls\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.975078 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.975054 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73c52f87-4e3e-4427-8112-f1ca2c2fba25-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.975196 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.975100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73c52f87-4e3e-4427-8112-f1ca2c2fba25-metrics-client-ca\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.975196 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.975148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/73c52f87-4e3e-4427-8112-f1ca2c2fba25-federate-client-tls\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.975196 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.975178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/73c52f87-4e3e-4427-8112-f1ca2c2fba25-secret-telemeter-client\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.975336 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.975225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73c52f87-4e3e-4427-8112-f1ca2c2fba25-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.975336 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.975300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73c52f87-4e3e-4427-8112-f1ca2c2fba25-serving-certs-ca-bundle\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.976097 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.976072 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73c52f87-4e3e-4427-8112-f1ca2c2fba25-metrics-client-ca\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.976791 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.976767 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73c52f87-4e3e-4427-8112-f1ca2c2fba25-serving-certs-ca-bundle\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.976946 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.976899 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73c52f87-4e3e-4427-8112-f1ca2c2fba25-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.978073 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.978039 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/73c52f87-4e3e-4427-8112-f1ca2c2fba25-telemeter-client-tls\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.978850 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.978812 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/73c52f87-4e3e-4427-8112-f1ca2c2fba25-federate-client-tls\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.979081 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.979041 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/73c52f87-4e3e-4427-8112-f1ca2c2fba25-secret-telemeter-client\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.979425 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.979407 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73c52f87-4e3e-4427-8112-f1ca2c2fba25-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:51.984381 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:51.984362 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x78mn\" (UniqueName: \"kubernetes.io/projected/73c52f87-4e3e-4427-8112-f1ca2c2fba25-kube-api-access-x78mn\") pod \"telemeter-client-6798878cfc-9kgql\" (UID: \"73c52f87-4e3e-4427-8112-f1ca2c2fba25\") " pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:52.059864 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:52.059825 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:52.065836 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:52.065807 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" Jul 17 21:28:52.423589 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:52.423486 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zz8sh" event={"ID":"31b165ae-1d35-4dfa-8ff9-f693f65bc628","Type":"ContainerStarted","Data":"21752130b0e13608abd82ad771883e55408173ff7eb68b1577930f803d4605cc"} Jul 17 21:28:52.423589 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:52.423565 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zz8sh" event={"ID":"31b165ae-1d35-4dfa-8ff9-f693f65bc628","Type":"ContainerStarted","Data":"bd4a1a7ab3e4147db412ff91bc6b33a7d7664a937ff5201fddc248a2fd99c83a"} Jul 17 21:28:52.443937 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:52.443871 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zz8sh" podStartSLOduration=2.691786265 podStartE2EDuration="6.443851412s" totalImagesPullingTime="3.651781229s" totalInitContainerRuntime="100.283918ms" isStatefulPod=true podCreationTimestamp="2026-07-17 21:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:52.442019919 +0000 UTC m=+73.863739170" watchObservedRunningTime="2026-07-17 21:28:52.443851412 +0000 UTC m=+73.865570644" Jul 17 21:28:52.645381 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:52.645349 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6798878cfc-9kgql"] Jul 17 21:28:52.668708 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:28:52.668675 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c52f87_4e3e_4427_8112_f1ca2c2fba25.slice/crio-5132980bab7fca09d09de8040d9058a13a7b9cc87e7d70cb96bdeae7c09b0c6a WatchSource:0}: Error finding container 5132980bab7fca09d09de8040d9058a13a7b9cc87e7d70cb96bdeae7c09b0c6a: Status 404 returned error can't find the container with id 5132980bab7fca09d09de8040d9058a13a7b9cc87e7d70cb96bdeae7c09b0c6a Jul 17 21:28:52.735621 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:52.735593 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:53.429833 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:53.429786 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" event={"ID":"8785431a-f6f7-4fc6-a97f-268e35ad79e9","Type":"ContainerStarted","Data":"2a921cebdc7db5ee8581cc4ece7df9122caa9c9b2c57fbea106fbfca55309f06"} Jul 17 21:28:53.430309 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:53.429878 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" event={"ID":"8785431a-f6f7-4fc6-a97f-268e35ad79e9","Type":"ContainerStarted","Data":"ce96eabe4c0fdaa15b5bbd576b8ff5dc1298a38373c6f791181f8beee9d6334b"} Jul 17 21:28:53.430309 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:53.429919 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" event={"ID":"8785431a-f6f7-4fc6-a97f-268e35ad79e9","Type":"ContainerStarted","Data":"74969a08b5112b99edfa9c91838835c71a2d77c586356d3c42724069cf63a335"} Jul 17 21:28:53.431715 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:53.431683 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" event={"ID":"73c52f87-4e3e-4427-8112-f1ca2c2fba25","Type":"ContainerStarted","Data":"5132980bab7fca09d09de8040d9058a13a7b9cc87e7d70cb96bdeae7c09b0c6a"} Jul 17 21:28:53.434890 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:53.433717 2580 generic.go:356] "Generic (PLEG): container finished" podID="aa077b26-a698-4b72-8f63-b8d3edf4d8ee" containerID="c0277d0eb1215fe0774c95cee7051782a841b8c7632c7d50ed545d8978376f80" exitCode=0 Jul 17 21:28:53.434890 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:53.434646 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa077b26-a698-4b72-8f63-b8d3edf4d8ee","Type":"ContainerDied","Data":"c0277d0eb1215fe0774c95cee7051782a841b8c7632c7d50ed545d8978376f80"} Jul 17 21:28:54.442144 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:54.442085 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" event={"ID":"8785431a-f6f7-4fc6-a97f-268e35ad79e9","Type":"ContainerStarted","Data":"7bb1545d0d5888db7412b6a6e0aae82304f64d6d2f40d38faf39891a62aa255c"} Jul 17 21:28:54.442668 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:54.442224 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" event={"ID":"8785431a-f6f7-4fc6-a97f-268e35ad79e9","Type":"ContainerStarted","Data":"1e03889d288463024e05c8fe8ca3ac2d2d2d12945333c46636a50bd440f78e64"} Jul 17 21:28:54.442668 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:54.442269 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" event={"ID":"8785431a-f6f7-4fc6-a97f-268e35ad79e9","Type":"ContainerStarted","Data":"32c6124a45906c3612f812f92676068755ac808d24e6245af108158555cbef5f"} Jul 17 21:28:54.442668 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:54.442527 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:28:54.465864 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:54.465779 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" podStartSLOduration=2.900855918 podStartE2EDuration="5.465762394s" totalImagesPullingTime="2.564906476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:54.463300792 +0000 UTC m=+75.885020044" watchObservedRunningTime="2026-07-17 21:28:54.465762394 +0000 UTC m=+75.887481633" Jul 17 21:28:54.598459 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:54.598419 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:54.698345 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:54.698260 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:55.349180 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:55.349154 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bc4774985-2r78g" Jul 17 21:28:55.448327 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:55.448288 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" event={"ID":"73c52f87-4e3e-4427-8112-f1ca2c2fba25","Type":"ContainerStarted","Data":"95bad95decd2eb3c40ee575ec402ecd27c2afbe0d9d4422107c3c3507a6d52fd"} Jul 17 21:28:55.448770 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:55.448363 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" event={"ID":"73c52f87-4e3e-4427-8112-f1ca2c2fba25","Type":"ContainerStarted","Data":"da92d933253228b79d646b5354e5272c61cb96f910f7a00019f72bb9ce472959"} Jul 17 21:28:55.448770 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:55.448410 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" event={"ID":"73c52f87-4e3e-4427-8112-f1ca2c2fba25","Type":"ContainerStarted","Data":"efd61e32f60d70c1c5c5a765fcb4495c3e6c76b71d3d21906e3fb0b6b6b70c72"} Jul 17 21:28:55.451846 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:55.451818 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa077b26-a698-4b72-8f63-b8d3edf4d8ee","Type":"ContainerStarted","Data":"d77bc22c25229e67e470ebda771b8ae84834e3cd2174e2febb7ddfde2d03b1fd"} Jul 17 21:28:55.451964 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:55.451897 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa077b26-a698-4b72-8f63-b8d3edf4d8ee","Type":"ContainerStarted","Data":"38717191e77831be4be0f7de69afd02dbaf1da5212681193ba33a5fdfec6a904"} Jul 17 21:28:55.451964 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:55.451951 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa077b26-a698-4b72-8f63-b8d3edf4d8ee","Type":"ContainerStarted","Data":"3af5553c215f26a284398fa2d03fc5522848b9f9ff986529e782aeda1890a76c"} Jul 17 21:28:55.452048 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:55.451980 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa077b26-a698-4b72-8f63-b8d3edf4d8ee","Type":"ContainerStarted","Data":"8d4841045a180070274a2c81e87b4adef6c893a32e0d6252f21868a1f3bfc0e6"} Jul 17 21:28:55.452048 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:55.452004 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa077b26-a698-4b72-8f63-b8d3edf4d8ee","Type":"ContainerStarted","Data":"49796a12bc3b6cb39572b007bc7d5318516e2ca856157c44d90d85bcff761047"} Jul 17 21:28:55.452048 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:55.452028 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa077b26-a698-4b72-8f63-b8d3edf4d8ee","Type":"ContainerStarted","Data":"b5086a3009bce98ef462c6f74f89b1ec5c38c17bd03fb214552bb1172b258134"} Jul 17 21:28:55.485582 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:55.485492 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6798878cfc-9kgql" podStartSLOduration=2.272167943 podStartE2EDuration="4.48547499s" totalImagesPullingTime="2.213307047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:55.483325071 +0000 UTC m=+76.905044338" watchObservedRunningTime="2026-07-17 21:28:55.48547499 +0000 UTC m=+76.907194238" Jul 17 21:28:55.563108 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:55.563080 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:55.663094 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:55.663058 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:55.764285 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:55.764260 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:55.808575 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:55.808541 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad6b74e2-eda2-430c-838a-5a702016c7bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:55.808740 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:55.808604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:55.809096 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:55.809076 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad6b74e2-eda2-430c-838a-5a702016c7bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:55.811160 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:55.811136 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad6b74e2-eda2-430c-838a-5a702016c7bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5d5nk\" (UID: \"ad6b74e2-eda2-430c-838a-5a702016c7bd\") " pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:55.873633 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:55.873598 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5d5nk" Jul 17 21:28:56.007329 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:56.007269 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.741988484 podStartE2EDuration="9.007248533s" totalImagesPullingTime="3.241546432s" totalInitContainerRuntime="23.713617ms" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:55.534411286 +0000 UTC m=+76.956130511" watchObservedRunningTime="2026-07-17 21:28:56.007248533 +0000 UTC m=+77.428967766" Jul 17 21:28:56.007521 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:56.007505 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5d5nk"] Jul 17 21:28:56.011362 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:28:56.011330 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad6b74e2_eda2_430c_838a_5a702016c7bd.slice/crio-88c116476fe37c0f3d435ad2dc3264c8156cfc51ebc4c6a641e4b802b7339394 WatchSource:0}: Error finding container 88c116476fe37c0f3d435ad2dc3264c8156cfc51ebc4c6a641e4b802b7339394: Status 404 returned error can't find the container with id 88c116476fe37c0f3d435ad2dc3264c8156cfc51ebc4c6a641e4b802b7339394 Jul 17 21:28:56.457862 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:56.457776 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5d5nk" event={"ID":"ad6b74e2-eda2-430c-838a-5a702016c7bd","Type":"ContainerStarted","Data":"88c116476fe37c0f3d435ad2dc3264c8156cfc51ebc4c6a641e4b802b7339394"} Jul 17 21:28:57.103472 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:57.103435 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:57.211698 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:57.211656 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:58.761749 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:58.761716 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:58.863431 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:58.863357 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:59.245375 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:59.245298 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:28:59.346429 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:28:59.346240 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:29:00.459001 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:29:00.458969 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-76bb7cbc89-5flq2" Jul 17 21:29:01.072733 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:29:01.072702 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:29:01.135412 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:29:01.135362 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:29:01.486594 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:29:01.486555 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5d5nk" event={"ID":"ad6b74e2-eda2-430c-838a-5a702016c7bd","Type":"ContainerStarted","Data":"bc6e1c5c9a3f54410c75143be856f7a7bd23020bba0bfca3daec219726d6560f"} Jul 17 21:29:04.500288 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:29:04.500256 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5d5nk" event={"ID":"ad6b74e2-eda2-430c-838a-5a702016c7bd","Type":"ContainerStarted","Data":"36fd1784f6014dc588eadc1f04c32978c92361f79a0228d2468ef06d43c77392"} Jul 17 21:29:06.943367 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:29:06.943330 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:29:07.051427 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:29:07.051368 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:29:07.138216 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:29:07.138183 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:29:08.525722 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:29:08.525687 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5d5nk" event={"ID":"ad6b74e2-eda2-430c-838a-5a702016c7bd","Type":"ContainerStarted","Data":"1abedebb191ec9bbdf270c9235edcbca62e32551a849da4dc67035796e7f9e26"} Jul 17 21:29:08.544570 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:29:08.544521 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5d5nk" podStartSLOduration=17.774383232 podStartE2EDuration="29.54450566s" totalImagesPullingTime="11.770122428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:28:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:29:08.54370538 +0000 UTC m=+89.965424632" watchObservedRunningTime="2026-07-17 21:29:08.54450566 +0000 UTC m=+89.966224894" Jul 17 21:29:16.621904 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:29:16.621869 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:29:16.687887 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:29:16.687856 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:29:20.261301 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:29:20.261266 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:29:20.319280 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:29:20.319243 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:29:22.426045 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:29:22.426014 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-n7c88" Jul 17 21:29:24.031684 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:29:24.031646 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:29:24.132275 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:29:24.132242 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:29:24.189826 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:29:24.189794 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:30:02.655820 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:30:02.655780 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:30:02.715804 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:30:02.715770 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:30:02.835264 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:30:02.835233 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:30:06.135310 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:30:06.135277 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:30:06.210297 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:30:06.210262 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:30:06.281679 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:30:06.281647 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:30:06.872490 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:30:06.872447 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:30:06.949156 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:30:06.949118 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:30:10.212572 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:30:10.212514 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:30:10.265136 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:30:10.265099 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:30:14.613723 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:30:14.613684 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:30:14.665634 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:30:14.665601 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:30:17.267242 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:30:17.267199 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:30:17.377749 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:30:17.377711 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:30:31.598244 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:30:31.598147 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:30:31.696355 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:30:31.696312 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:31:02.556873 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:02.556834 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:31:02.617109 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:02.617076 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:31:26.359997 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:26.359964 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:31:26.460145 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:26.460089 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:31:27.023171 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:27.023138 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk"] Jul 17 21:31:27.026464 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:27.026444 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk" Jul 17 21:31:27.029893 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:27.029861 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 17 21:31:27.030982 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:27.030953 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 17 21:31:27.031440 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:27.031419 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7qg4z\"" Jul 17 21:31:27.035539 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:27.035427 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk"] Jul 17 21:31:27.119859 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:27.119821 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf27a349-9a23-4d07-9ce0-86743ee6e147-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk\" (UID: \"bf27a349-9a23-4d07-9ce0-86743ee6e147\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk" Jul 17 21:31:27.119859 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:27.119862 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j8hw\" (UniqueName: \"kubernetes.io/projected/bf27a349-9a23-4d07-9ce0-86743ee6e147-kube-api-access-5j8hw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk\" (UID: \"bf27a349-9a23-4d07-9ce0-86743ee6e147\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk" Jul 17 21:31:27.120076 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:27.119895 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf27a349-9a23-4d07-9ce0-86743ee6e147-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk\" (UID: \"bf27a349-9a23-4d07-9ce0-86743ee6e147\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk" Jul 17 21:31:27.220644 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:27.220603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j8hw\" (UniqueName: \"kubernetes.io/projected/bf27a349-9a23-4d07-9ce0-86743ee6e147-kube-api-access-5j8hw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk\" (UID: \"bf27a349-9a23-4d07-9ce0-86743ee6e147\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk" Jul 17 21:31:27.220644 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:27.220647 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf27a349-9a23-4d07-9ce0-86743ee6e147-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk\" (UID: \"bf27a349-9a23-4d07-9ce0-86743ee6e147\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk" Jul 17 21:31:27.220849 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:27.220717 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf27a349-9a23-4d07-9ce0-86743ee6e147-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk\" (UID: \"bf27a349-9a23-4d07-9ce0-86743ee6e147\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk" Jul 17 21:31:27.221044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:27.221029 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf27a349-9a23-4d07-9ce0-86743ee6e147-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk\" (UID: \"bf27a349-9a23-4d07-9ce0-86743ee6e147\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk" Jul 17 21:31:27.221109 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:27.221089 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf27a349-9a23-4d07-9ce0-86743ee6e147-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk\" (UID: \"bf27a349-9a23-4d07-9ce0-86743ee6e147\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk" Jul 17 21:31:27.230594 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:27.230564 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j8hw\" (UniqueName: \"kubernetes.io/projected/bf27a349-9a23-4d07-9ce0-86743ee6e147-kube-api-access-5j8hw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk\" (UID: \"bf27a349-9a23-4d07-9ce0-86743ee6e147\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk" Jul 17 21:31:27.341239 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:27.341142 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk" Jul 17 21:31:27.471993 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:27.471897 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk"] Jul 17 21:31:27.475163 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:31:27.475139 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf27a349_9a23_4d07_9ce0_86743ee6e147.slice/crio-aac714f420688667f68ddaa2cc5b38ba3eceeb94fa4e55c92f2eb198560d408d WatchSource:0}: Error finding container aac714f420688667f68ddaa2cc5b38ba3eceeb94fa4e55c92f2eb198560d408d: Status 404 returned error can't find the container with id aac714f420688667f68ddaa2cc5b38ba3eceeb94fa4e55c92f2eb198560d408d Jul 17 21:31:28.034668 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:28.034633 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk" event={"ID":"bf27a349-9a23-4d07-9ce0-86743ee6e147","Type":"ContainerStarted","Data":"aac714f420688667f68ddaa2cc5b38ba3eceeb94fa4e55c92f2eb198560d408d"} Jul 17 21:31:28.985600 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:28.985567 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:31:35.064536 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:35.064500 2580 generic.go:356] "Generic (PLEG): container finished" podID="bf27a349-9a23-4d07-9ce0-86743ee6e147" containerID="d5be85fa4df22bb76966a0bd33e331fa958c37c24542dcc3d5ab3e71f18b35f4" exitCode=0 Jul 17 21:31:35.064966 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:35.064559 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk" event={"ID":"bf27a349-9a23-4d07-9ce0-86743ee6e147","Type":"ContainerDied","Data":"d5be85fa4df22bb76966a0bd33e331fa958c37c24542dcc3d5ab3e71f18b35f4"} Jul 17 21:31:38.078132 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:38.078088 2580 generic.go:356] "Generic (PLEG): container finished" podID="bf27a349-9a23-4d07-9ce0-86743ee6e147" containerID="39a669ff07a62adb14a76cd191371731ac2c478fdbe482a763d9fbba8ca6900d" exitCode=0 Jul 17 21:31:38.078543 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:38.078167 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk" event={"ID":"bf27a349-9a23-4d07-9ce0-86743ee6e147","Type":"ContainerDied","Data":"39a669ff07a62adb14a76cd191371731ac2c478fdbe482a763d9fbba8ca6900d"} Jul 17 21:31:46.111376 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:46.111334 2580 generic.go:356] "Generic (PLEG): container finished" podID="bf27a349-9a23-4d07-9ce0-86743ee6e147" containerID="95f8dbc1e28f8a40ba0ae2cf356f8ae8a76be07bc9869de84fbc5266948a81e4" exitCode=0 Jul 17 21:31:46.111833 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:46.111431 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk" event={"ID":"bf27a349-9a23-4d07-9ce0-86743ee6e147","Type":"ContainerDied","Data":"95f8dbc1e28f8a40ba0ae2cf356f8ae8a76be07bc9869de84fbc5266948a81e4"} Jul 17 21:31:47.238685 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:47.238657 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk" Jul 17 21:31:47.401262 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:47.401167 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf27a349-9a23-4d07-9ce0-86743ee6e147-bundle\") pod \"bf27a349-9a23-4d07-9ce0-86743ee6e147\" (UID: \"bf27a349-9a23-4d07-9ce0-86743ee6e147\") " Jul 17 21:31:47.401451 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:47.401264 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5j8hw\" (UniqueName: \"kubernetes.io/projected/bf27a349-9a23-4d07-9ce0-86743ee6e147-kube-api-access-5j8hw\") pod \"bf27a349-9a23-4d07-9ce0-86743ee6e147\" (UID: \"bf27a349-9a23-4d07-9ce0-86743ee6e147\") " Jul 17 21:31:47.401451 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:47.401310 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf27a349-9a23-4d07-9ce0-86743ee6e147-util\") pod \"bf27a349-9a23-4d07-9ce0-86743ee6e147\" (UID: \"bf27a349-9a23-4d07-9ce0-86743ee6e147\") " Jul 17 21:31:47.401853 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:47.401823 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf27a349-9a23-4d07-9ce0-86743ee6e147-bundle" (OuterVolumeSpecName: "bundle") pod "bf27a349-9a23-4d07-9ce0-86743ee6e147" (UID: "bf27a349-9a23-4d07-9ce0-86743ee6e147"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:31:47.403787 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:47.403756 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf27a349-9a23-4d07-9ce0-86743ee6e147-kube-api-access-5j8hw" (OuterVolumeSpecName: "kube-api-access-5j8hw") pod "bf27a349-9a23-4d07-9ce0-86743ee6e147" (UID: "bf27a349-9a23-4d07-9ce0-86743ee6e147"). InnerVolumeSpecName "kube-api-access-5j8hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:31:47.405763 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:47.405743 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf27a349-9a23-4d07-9ce0-86743ee6e147-util" (OuterVolumeSpecName: "util") pod "bf27a349-9a23-4d07-9ce0-86743ee6e147" (UID: "bf27a349-9a23-4d07-9ce0-86743ee6e147"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:31:47.502716 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:47.502685 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf27a349-9a23-4d07-9ce0-86743ee6e147-bundle\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:31:47.502716 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:47.502712 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j8hw\" (UniqueName: \"kubernetes.io/projected/bf27a349-9a23-4d07-9ce0-86743ee6e147-kube-api-access-5j8hw\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:31:47.502716 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:47.502723 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf27a349-9a23-4d07-9ce0-86743ee6e147-util\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:31:48.120876 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:48.120834 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk" event={"ID":"bf27a349-9a23-4d07-9ce0-86743ee6e147","Type":"ContainerDied","Data":"aac714f420688667f68ddaa2cc5b38ba3eceeb94fa4e55c92f2eb198560d408d"} Jul 17 21:31:48.120876 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:48.120867 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lv2qk" Jul 17 21:31:48.120876 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:48.120881 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aac714f420688667f68ddaa2cc5b38ba3eceeb94fa4e55c92f2eb198560d408d" Jul 17 21:31:53.090595 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:53.090558 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:31:54.071960 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:54.071915 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:31:55.042699 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.042666 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vf74r"] Jul 17 21:31:55.043066 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.042949 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf27a349-9a23-4d07-9ce0-86743ee6e147" containerName="extract" Jul 17 21:31:55.043066 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.042960 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf27a349-9a23-4d07-9ce0-86743ee6e147" containerName="extract" Jul 17 21:31:55.043066 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.042979 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf27a349-9a23-4d07-9ce0-86743ee6e147" containerName="util" Jul 17 21:31:55.043066 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.042987 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf27a349-9a23-4d07-9ce0-86743ee6e147" containerName="util" Jul 17 21:31:55.043066 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.043004 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf27a349-9a23-4d07-9ce0-86743ee6e147" containerName="pull" Jul 17 21:31:55.043066 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.043008 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf27a349-9a23-4d07-9ce0-86743ee6e147" containerName="pull" Jul 17 21:31:55.043066 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.043054 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf27a349-9a23-4d07-9ce0-86743ee6e147" containerName="extract" Jul 17 21:31:55.046142 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.046126 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vf74r" Jul 17 21:31:55.048777 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.048753 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 17 21:31:55.049087 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.049070 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-svznj\"" Jul 17 21:31:55.049340 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.049327 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 17 21:31:55.060218 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.060196 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vf74r"] Jul 17 21:31:55.162465 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.162427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d124adbb-115e-41bd-987c-c32e0e3f201d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vf74r\" (UID: \"d124adbb-115e-41bd-987c-c32e0e3f201d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vf74r" Jul 17 21:31:55.162659 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.162475 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6jks\" (UniqueName: \"kubernetes.io/projected/d124adbb-115e-41bd-987c-c32e0e3f201d-kube-api-access-h6jks\") pod \"cert-manager-operator-controller-manager-55c444bf86-vf74r\" (UID: \"d124adbb-115e-41bd-987c-c32e0e3f201d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vf74r" Jul 17 21:31:55.263822 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.263785 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d124adbb-115e-41bd-987c-c32e0e3f201d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vf74r\" (UID: \"d124adbb-115e-41bd-987c-c32e0e3f201d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vf74r" Jul 17 21:31:55.263822 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.263825 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6jks\" (UniqueName: \"kubernetes.io/projected/d124adbb-115e-41bd-987c-c32e0e3f201d-kube-api-access-h6jks\") pod \"cert-manager-operator-controller-manager-55c444bf86-vf74r\" (UID: \"d124adbb-115e-41bd-987c-c32e0e3f201d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vf74r" Jul 17 21:31:55.264174 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.264156 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d124adbb-115e-41bd-987c-c32e0e3f201d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vf74r\" (UID: \"d124adbb-115e-41bd-987c-c32e0e3f201d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vf74r" Jul 17 21:31:55.274215 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.274181 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6jks\" (UniqueName: \"kubernetes.io/projected/d124adbb-115e-41bd-987c-c32e0e3f201d-kube-api-access-h6jks\") pod \"cert-manager-operator-controller-manager-55c444bf86-vf74r\" (UID: \"d124adbb-115e-41bd-987c-c32e0e3f201d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vf74r" Jul 17 21:31:55.355332 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.355245 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vf74r" Jul 17 21:31:55.492762 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:55.492692 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vf74r"] Jul 17 21:31:55.495427 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:31:55.495377 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd124adbb_115e_41bd_987c_c32e0e3f201d.slice/crio-7412384fa9b7af254b104f0236456cbffbfa3ea68ea4afe6fcb174f88fd529d6 WatchSource:0}: Error finding container 7412384fa9b7af254b104f0236456cbffbfa3ea68ea4afe6fcb174f88fd529d6: Status 404 returned error can't find the container with id 7412384fa9b7af254b104f0236456cbffbfa3ea68ea4afe6fcb174f88fd529d6 Jul 17 21:31:56.150702 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:56.150671 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vf74r" event={"ID":"d124adbb-115e-41bd-987c-c32e0e3f201d","Type":"ContainerStarted","Data":"7412384fa9b7af254b104f0236456cbffbfa3ea68ea4afe6fcb174f88fd529d6"} Jul 17 21:31:56.711650 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:56.711620 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:31:56.810760 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:56.810726 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:31:57.157667 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:57.157635 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vf74r" event={"ID":"d124adbb-115e-41bd-987c-c32e0e3f201d","Type":"ContainerStarted","Data":"99c91259b855f4b7d22782bb69de7c0b6ef38eaed3a0caf4932e07e23a449d79"} Jul 17 21:31:57.185108 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:57.185048 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vf74r" podStartSLOduration=0.616154377 podStartE2EDuration="2.185026332s" totalImagesPullingTime="1.568871955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:31:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:31:57.184370486 +0000 UTC m=+258.606089739" watchObservedRunningTime="2026-07-17 21:31:57.185026332 +0000 UTC m=+258.606745566" Jul 17 21:31:57.684315 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:57.684273 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:31:57.809044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:57.808999 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:31:58.007602 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.007514 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7vbqh"] Jul 17 21:31:58.009517 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.009503 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7vbqh" Jul 17 21:31:58.013241 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.013215 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 17 21:31:58.013361 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.013318 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 17 21:31:58.026706 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.026683 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7vbqh"] Jul 17 21:31:58.189200 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.189165 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpp54\" (UniqueName: \"kubernetes.io/projected/f74c90e5-7519-4933-9a11-18a10ee2d992-kube-api-access-tpp54\") pod \"cert-manager-webhook-6486ffd997-7vbqh\" (UID: \"f74c90e5-7519-4933-9a11-18a10ee2d992\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7vbqh" Jul 17 21:31:58.189581 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.189216 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f74c90e5-7519-4933-9a11-18a10ee2d992-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7vbqh\" (UID: \"f74c90e5-7519-4933-9a11-18a10ee2d992\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7vbqh" Jul 17 21:31:58.290265 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.290223 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f74c90e5-7519-4933-9a11-18a10ee2d992-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7vbqh\" (UID: \"f74c90e5-7519-4933-9a11-18a10ee2d992\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7vbqh" Jul 17 21:31:58.290446 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.290315 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpp54\" (UniqueName: \"kubernetes.io/projected/f74c90e5-7519-4933-9a11-18a10ee2d992-kube-api-access-tpp54\") pod \"cert-manager-webhook-6486ffd997-7vbqh\" (UID: \"f74c90e5-7519-4933-9a11-18a10ee2d992\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7vbqh" Jul 17 21:31:58.299570 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.299534 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f74c90e5-7519-4933-9a11-18a10ee2d992-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7vbqh\" (UID: \"f74c90e5-7519-4933-9a11-18a10ee2d992\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7vbqh" Jul 17 21:31:58.299694 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.299622 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpp54\" (UniqueName: \"kubernetes.io/projected/f74c90e5-7519-4933-9a11-18a10ee2d992-kube-api-access-tpp54\") pod \"cert-manager-webhook-6486ffd997-7vbqh\" (UID: \"f74c90e5-7519-4933-9a11-18a10ee2d992\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7vbqh" Jul 17 21:31:58.331384 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.331353 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7vbqh" Jul 17 21:31:58.461810 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.461789 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7vbqh"] Jul 17 21:31:58.464257 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:31:58.464226 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf74c90e5_7519_4933_9a11_18a10ee2d992.slice/crio-b089a46198656d4870da74372362ca9d110f2282c415fa98a0e9061059ca6c54 WatchSource:0}: Error finding container b089a46198656d4870da74372362ca9d110f2282c415fa98a0e9061059ca6c54: Status 404 returned error can't find the container with id b089a46198656d4870da74372362ca9d110f2282c415fa98a0e9061059ca6c54 Jul 17 21:31:58.516078 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.516050 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c"] Jul 17 21:31:58.519155 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.519140 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c" Jul 17 21:31:58.521709 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.521687 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 17 21:31:58.521818 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.521710 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 17 21:31:58.521818 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.521699 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7qg4z\"" Jul 17 21:31:58.527332 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.527312 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c"] Jul 17 21:31:58.694657 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.694560 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf8dc401-c0b8-49fe-9283-e0d18ba38a70-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c\" (UID: \"bf8dc401-c0b8-49fe-9283-e0d18ba38a70\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c" Jul 17 21:31:58.694657 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.694621 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn7dm\" (UniqueName: \"kubernetes.io/projected/bf8dc401-c0b8-49fe-9283-e0d18ba38a70-kube-api-access-zn7dm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c\" (UID: \"bf8dc401-c0b8-49fe-9283-e0d18ba38a70\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c" Jul 17 21:31:58.694860 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.694668 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf8dc401-c0b8-49fe-9283-e0d18ba38a70-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c\" (UID: \"bf8dc401-c0b8-49fe-9283-e0d18ba38a70\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c" Jul 17 21:31:58.795378 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.795338 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf8dc401-c0b8-49fe-9283-e0d18ba38a70-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c\" (UID: \"bf8dc401-c0b8-49fe-9283-e0d18ba38a70\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c" Jul 17 21:31:58.795595 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.795471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf8dc401-c0b8-49fe-9283-e0d18ba38a70-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c\" (UID: \"bf8dc401-c0b8-49fe-9283-e0d18ba38a70\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c" Jul 17 21:31:58.795595 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.795504 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn7dm\" (UniqueName: \"kubernetes.io/projected/bf8dc401-c0b8-49fe-9283-e0d18ba38a70-kube-api-access-zn7dm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c\" (UID: \"bf8dc401-c0b8-49fe-9283-e0d18ba38a70\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c" Jul 17 21:31:58.795751 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.795731 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf8dc401-c0b8-49fe-9283-e0d18ba38a70-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c\" (UID: \"bf8dc401-c0b8-49fe-9283-e0d18ba38a70\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c" Jul 17 21:31:58.795801 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.795766 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf8dc401-c0b8-49fe-9283-e0d18ba38a70-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c\" (UID: \"bf8dc401-c0b8-49fe-9283-e0d18ba38a70\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c" Jul 17 21:31:58.804667 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.804649 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn7dm\" (UniqueName: \"kubernetes.io/projected/bf8dc401-c0b8-49fe-9283-e0d18ba38a70-kube-api-access-zn7dm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c\" (UID: \"bf8dc401-c0b8-49fe-9283-e0d18ba38a70\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c" Jul 17 21:31:58.829243 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.829217 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c" Jul 17 21:31:58.957920 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:58.957889 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c"] Jul 17 21:31:58.960369 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:31:58.960340 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8dc401_c0b8_49fe_9283_e0d18ba38a70.slice/crio-94ab4e679dd895d94afc17777b6419d51cdf607a134cc15ab276bfa2b20bbc41 WatchSource:0}: Error finding container 94ab4e679dd895d94afc17777b6419d51cdf607a134cc15ab276bfa2b20bbc41: Status 404 returned error can't find the container with id 94ab4e679dd895d94afc17777b6419d51cdf607a134cc15ab276bfa2b20bbc41 Jul 17 21:31:59.166298 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:59.166257 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7vbqh" event={"ID":"f74c90e5-7519-4933-9a11-18a10ee2d992","Type":"ContainerStarted","Data":"b089a46198656d4870da74372362ca9d110f2282c415fa98a0e9061059ca6c54"} Jul 17 21:31:59.167694 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:59.167669 2580 generic.go:356] "Generic (PLEG): container finished" podID="bf8dc401-c0b8-49fe-9283-e0d18ba38a70" containerID="1ab9c9654000e36546537cc6e7eddd1ce9260fa76a7d9f0358aa12f778ce6446" exitCode=0 Jul 17 21:31:59.167806 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:59.167706 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c" event={"ID":"bf8dc401-c0b8-49fe-9283-e0d18ba38a70","Type":"ContainerDied","Data":"1ab9c9654000e36546537cc6e7eddd1ce9260fa76a7d9f0358aa12f778ce6446"} Jul 17 21:31:59.167806 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:31:59.167741 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c" event={"ID":"bf8dc401-c0b8-49fe-9283-e0d18ba38a70","Type":"ContainerStarted","Data":"94ab4e679dd895d94afc17777b6419d51cdf607a134cc15ab276bfa2b20bbc41"} Jul 17 21:32:00.131709 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:00.131604 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:00.346653 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:00.346617 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:01.462417 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:01.462311 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qmhkx"] Jul 17 21:32:01.465358 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:01.465343 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmhkx" Jul 17 21:32:01.467729 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:01.467710 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8mrnp\"" Jul 17 21:32:01.475367 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:01.475346 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qmhkx"] Jul 17 21:32:01.519948 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:01.519914 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05ac8b72-1973-4d4c-bc1b-2e9ab620ad04-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qmhkx\" (UID: \"05ac8b72-1973-4d4c-bc1b-2e9ab620ad04\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmhkx" Jul 17 21:32:01.520113 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:01.519969 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8b74\" (UniqueName: \"kubernetes.io/projected/05ac8b72-1973-4d4c-bc1b-2e9ab620ad04-kube-api-access-d8b74\") pod \"cert-manager-cainjector-559cdd8c87-qmhkx\" (UID: \"05ac8b72-1973-4d4c-bc1b-2e9ab620ad04\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmhkx" Jul 17 21:32:01.620886 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:01.620850 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05ac8b72-1973-4d4c-bc1b-2e9ab620ad04-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qmhkx\" (UID: \"05ac8b72-1973-4d4c-bc1b-2e9ab620ad04\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmhkx" Jul 17 21:32:01.621058 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:01.620916 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8b74\" (UniqueName: \"kubernetes.io/projected/05ac8b72-1973-4d4c-bc1b-2e9ab620ad04-kube-api-access-d8b74\") pod \"cert-manager-cainjector-559cdd8c87-qmhkx\" (UID: \"05ac8b72-1973-4d4c-bc1b-2e9ab620ad04\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmhkx" Jul 17 21:32:01.629267 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:01.629246 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05ac8b72-1973-4d4c-bc1b-2e9ab620ad04-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qmhkx\" (UID: \"05ac8b72-1973-4d4c-bc1b-2e9ab620ad04\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmhkx" Jul 17 21:32:01.629422 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:01.629370 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8b74\" (UniqueName: \"kubernetes.io/projected/05ac8b72-1973-4d4c-bc1b-2e9ab620ad04-kube-api-access-d8b74\") pod \"cert-manager-cainjector-559cdd8c87-qmhkx\" (UID: \"05ac8b72-1973-4d4c-bc1b-2e9ab620ad04\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmhkx" Jul 17 21:32:01.773795 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:01.773765 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmhkx" Jul 17 21:32:01.900180 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:01.900148 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qmhkx"] Jul 17 21:32:01.902962 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:32:01.902931 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05ac8b72_1973_4d4c_bc1b_2e9ab620ad04.slice/crio-02108497e7378bc5acc8cb469131fa85b0a70edde794c8c0cacb35fc645d9edc WatchSource:0}: Error finding container 02108497e7378bc5acc8cb469131fa85b0a70edde794c8c0cacb35fc645d9edc: Status 404 returned error can't find the container with id 02108497e7378bc5acc8cb469131fa85b0a70edde794c8c0cacb35fc645d9edc Jul 17 21:32:02.182291 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:02.182184 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmhkx" event={"ID":"05ac8b72-1973-4d4c-bc1b-2e9ab620ad04","Type":"ContainerStarted","Data":"02108497e7378bc5acc8cb469131fa85b0a70edde794c8c0cacb35fc645d9edc"} Jul 17 21:32:02.183987 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:02.183963 2580 generic.go:356] "Generic (PLEG): container finished" podID="bf8dc401-c0b8-49fe-9283-e0d18ba38a70" containerID="b4b36678e3d387cb28ef7badca354862cbf899e8e34ee901ea060f02d25461af" exitCode=0 Jul 17 21:32:02.184094 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:02.184009 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c" event={"ID":"bf8dc401-c0b8-49fe-9283-e0d18ba38a70","Type":"ContainerDied","Data":"b4b36678e3d387cb28ef7badca354862cbf899e8e34ee901ea060f02d25461af"} Jul 17 21:32:03.190625 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:03.190586 2580 generic.go:356] "Generic (PLEG): container finished" podID="bf8dc401-c0b8-49fe-9283-e0d18ba38a70" containerID="80237a4feab32c353cd24494076f065070bf3230a79cb12a2c5a4ccaafab3403" exitCode=0 Jul 17 21:32:03.191015 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:03.190628 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c" event={"ID":"bf8dc401-c0b8-49fe-9283-e0d18ba38a70","Type":"ContainerDied","Data":"80237a4feab32c353cd24494076f065070bf3230a79cb12a2c5a4ccaafab3403"} Jul 17 21:32:03.302743 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:03.302699 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:03.363048 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:03.363004 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:05.169886 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:05.169862 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c" Jul 17 21:32:05.202932 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:05.202896 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c" event={"ID":"bf8dc401-c0b8-49fe-9283-e0d18ba38a70","Type":"ContainerDied","Data":"94ab4e679dd895d94afc17777b6419d51cdf607a134cc15ab276bfa2b20bbc41"} Jul 17 21:32:05.203106 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:05.202954 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94ab4e679dd895d94afc17777b6419d51cdf607a134cc15ab276bfa2b20bbc41" Jul 17 21:32:05.203106 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:05.202912 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpgr8c" Jul 17 21:32:05.245219 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:05.245190 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf8dc401-c0b8-49fe-9283-e0d18ba38a70-util\") pod \"bf8dc401-c0b8-49fe-9283-e0d18ba38a70\" (UID: \"bf8dc401-c0b8-49fe-9283-e0d18ba38a70\") " Jul 17 21:32:05.245381 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:05.245245 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf8dc401-c0b8-49fe-9283-e0d18ba38a70-bundle\") pod \"bf8dc401-c0b8-49fe-9283-e0d18ba38a70\" (UID: \"bf8dc401-c0b8-49fe-9283-e0d18ba38a70\") " Jul 17 21:32:05.245381 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:05.245263 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn7dm\" (UniqueName: \"kubernetes.io/projected/bf8dc401-c0b8-49fe-9283-e0d18ba38a70-kube-api-access-zn7dm\") pod \"bf8dc401-c0b8-49fe-9283-e0d18ba38a70\" (UID: \"bf8dc401-c0b8-49fe-9283-e0d18ba38a70\") " Jul 17 21:32:05.245610 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:05.245587 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8dc401-c0b8-49fe-9283-e0d18ba38a70-bundle" (OuterVolumeSpecName: "bundle") pod "bf8dc401-c0b8-49fe-9283-e0d18ba38a70" (UID: "bf8dc401-c0b8-49fe-9283-e0d18ba38a70"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:32:05.247544 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:05.247513 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8dc401-c0b8-49fe-9283-e0d18ba38a70-kube-api-access-zn7dm" (OuterVolumeSpecName: "kube-api-access-zn7dm") pod "bf8dc401-c0b8-49fe-9283-e0d18ba38a70" (UID: "bf8dc401-c0b8-49fe-9283-e0d18ba38a70"). InnerVolumeSpecName "kube-api-access-zn7dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:32:05.250066 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:05.250028 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8dc401-c0b8-49fe-9283-e0d18ba38a70-util" (OuterVolumeSpecName: "util") pod "bf8dc401-c0b8-49fe-9283-e0d18ba38a70" (UID: "bf8dc401-c0b8-49fe-9283-e0d18ba38a70"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:32:05.345774 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:05.345737 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf8dc401-c0b8-49fe-9283-e0d18ba38a70-util\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:32:05.345774 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:05.345764 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf8dc401-c0b8-49fe-9283-e0d18ba38a70-bundle\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:32:05.345774 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:05.345773 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn7dm\" (UniqueName: \"kubernetes.io/projected/bf8dc401-c0b8-49fe-9283-e0d18ba38a70-kube-api-access-zn7dm\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:32:06.208756 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:06.208725 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmhkx" event={"ID":"05ac8b72-1973-4d4c-bc1b-2e9ab620ad04","Type":"ContainerStarted","Data":"6a68d28891a63028ea60b52c71e84f6f9ea074a68868f97e15c0f83749ebe01f"} Jul 17 21:32:06.225491 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:06.225447 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qmhkx" podStartSLOduration=1.9171893679999998 podStartE2EDuration="5.225429655s" totalImagesPullingTime="3.308240287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:32:06.224794182 +0000 UTC m=+267.646513465" watchObservedRunningTime="2026-07-17 21:32:06.225429655 +0000 UTC m=+267.647148887" Jul 17 21:32:07.520603 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:07.520575 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:09.712826 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:09.712792 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:09.813519 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:09.813488 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:17.223657 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:17.223622 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:17.260144 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:17.260109 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7vbqh" event={"ID":"f74c90e5-7519-4933-9a11-18a10ee2d992","Type":"ContainerStarted","Data":"db1e0ffb119379b1b946c97db4a76b4745ca18ec14771d0e1dbaaa627de38129"} Jul 17 21:32:17.260323 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:17.260307 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7vbqh" Jul 17 21:32:17.284208 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:17.284163 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-7vbqh" podStartSLOduration=1.6300487970000002 podStartE2EDuration="20.284148764s" totalImagesPullingTime="18.654099967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:32:17.283672388 +0000 UTC m=+278.705391622" watchObservedRunningTime="2026-07-17 21:32:17.284148764 +0000 UTC m=+278.705868008" Jul 17 21:32:17.294845 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:17.294814 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:17.421029 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:17.420938 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:17.503895 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:17.503864 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:19.030982 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:19.030952 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:21.771343 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.771298 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7"] Jul 17 21:32:21.771745 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.771729 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf8dc401-c0b8-49fe-9283-e0d18ba38a70" containerName="util" Jul 17 21:32:21.771787 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.771747 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf8dc401-c0b8-49fe-9283-e0d18ba38a70" containerName="util" Jul 17 21:32:21.771787 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.771755 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf8dc401-c0b8-49fe-9283-e0d18ba38a70" containerName="pull" Jul 17 21:32:21.771787 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.771760 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf8dc401-c0b8-49fe-9283-e0d18ba38a70" containerName="pull" Jul 17 21:32:21.771787 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.771770 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf8dc401-c0b8-49fe-9283-e0d18ba38a70" containerName="extract" Jul 17 21:32:21.771787 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.771775 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf8dc401-c0b8-49fe-9283-e0d18ba38a70" containerName="extract" Jul 17 21:32:21.771966 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.771824 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf8dc401-c0b8-49fe-9283-e0d18ba38a70" containerName="extract" Jul 17 21:32:21.773791 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.773775 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7" Jul 17 21:32:21.776319 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.776299 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 17 21:32:21.776319 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.776310 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 17 21:32:21.777344 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.777321 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7qg4z\"" Jul 17 21:32:21.782269 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.782249 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7"] Jul 17 21:32:21.872724 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.872691 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37a059fa-56e6-4f2e-98e2-1c79dfa639a2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7\" (UID: \"37a059fa-56e6-4f2e-98e2-1c79dfa639a2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7" Jul 17 21:32:21.872724 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.872724 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dlg8\" (UniqueName: \"kubernetes.io/projected/37a059fa-56e6-4f2e-98e2-1c79dfa639a2-kube-api-access-9dlg8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7\" (UID: \"37a059fa-56e6-4f2e-98e2-1c79dfa639a2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7" Jul 17 21:32:21.872925 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.872746 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37a059fa-56e6-4f2e-98e2-1c79dfa639a2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7\" (UID: \"37a059fa-56e6-4f2e-98e2-1c79dfa639a2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7" Jul 17 21:32:21.973968 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.973914 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37a059fa-56e6-4f2e-98e2-1c79dfa639a2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7\" (UID: \"37a059fa-56e6-4f2e-98e2-1c79dfa639a2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7" Jul 17 21:32:21.973968 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.973971 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dlg8\" (UniqueName: \"kubernetes.io/projected/37a059fa-56e6-4f2e-98e2-1c79dfa639a2-kube-api-access-9dlg8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7\" (UID: \"37a059fa-56e6-4f2e-98e2-1c79dfa639a2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7" Jul 17 21:32:21.974178 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.973995 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37a059fa-56e6-4f2e-98e2-1c79dfa639a2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7\" (UID: \"37a059fa-56e6-4f2e-98e2-1c79dfa639a2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7" Jul 17 21:32:21.974324 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.974305 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37a059fa-56e6-4f2e-98e2-1c79dfa639a2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7\" (UID: \"37a059fa-56e6-4f2e-98e2-1c79dfa639a2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7" Jul 17 21:32:21.974368 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.974333 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37a059fa-56e6-4f2e-98e2-1c79dfa639a2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7\" (UID: \"37a059fa-56e6-4f2e-98e2-1c79dfa639a2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7" Jul 17 21:32:21.984210 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:21.984179 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dlg8\" (UniqueName: \"kubernetes.io/projected/37a059fa-56e6-4f2e-98e2-1c79dfa639a2-kube-api-access-9dlg8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7\" (UID: \"37a059fa-56e6-4f2e-98e2-1c79dfa639a2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7" Jul 17 21:32:22.083642 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:22.083537 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7" Jul 17 21:32:22.224962 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:22.224936 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7"] Jul 17 21:32:22.227375 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:32:22.227351 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a059fa_56e6_4f2e_98e2_1c79dfa639a2.slice/crio-af4c0825ec21a1b65a49214e9138c6821528e47740c2d0fe67188d7bf9907a3e WatchSource:0}: Error finding container af4c0825ec21a1b65a49214e9138c6821528e47740c2d0fe67188d7bf9907a3e: Status 404 returned error can't find the container with id af4c0825ec21a1b65a49214e9138c6821528e47740c2d0fe67188d7bf9907a3e Jul 17 21:32:22.283112 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:22.283087 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7" event={"ID":"37a059fa-56e6-4f2e-98e2-1c79dfa639a2","Type":"ContainerStarted","Data":"af4c0825ec21a1b65a49214e9138c6821528e47740c2d0fe67188d7bf9907a3e"} Jul 17 21:32:23.267248 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:23.267219 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7vbqh" Jul 17 21:32:23.288974 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:23.288941 2580 generic.go:356] "Generic (PLEG): container finished" podID="37a059fa-56e6-4f2e-98e2-1c79dfa639a2" containerID="cb492ba30eb74505fa1301a3408c3dcc8d6732c86247e3599e3c6870f590d0e0" exitCode=0 Jul 17 21:32:23.289152 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:23.289023 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7" event={"ID":"37a059fa-56e6-4f2e-98e2-1c79dfa639a2","Type":"ContainerDied","Data":"cb492ba30eb74505fa1301a3408c3dcc8d6732c86247e3599e3c6870f590d0e0"} Jul 17 21:32:24.294863 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:24.294830 2580 generic.go:356] "Generic (PLEG): container finished" podID="37a059fa-56e6-4f2e-98e2-1c79dfa639a2" containerID="9a25a6ff92669455262c6a44b4cdab9456e2c6cfaa78c600016b5a062643a63a" exitCode=0 Jul 17 21:32:24.295248 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:24.294888 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7" event={"ID":"37a059fa-56e6-4f2e-98e2-1c79dfa639a2","Type":"ContainerDied","Data":"9a25a6ff92669455262c6a44b4cdab9456e2c6cfaa78c600016b5a062643a63a"} Jul 17 21:32:25.300953 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:25.300922 2580 generic.go:356] "Generic (PLEG): container finished" podID="37a059fa-56e6-4f2e-98e2-1c79dfa639a2" containerID="47ef1da56c3b68dbacbaa4b1a7bc301f1e71936230aa123f87f85a45c3f11bb3" exitCode=0 Jul 17 21:32:25.301352 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:25.301113 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7" event={"ID":"37a059fa-56e6-4f2e-98e2-1c79dfa639a2","Type":"ContainerDied","Data":"47ef1da56c3b68dbacbaa4b1a7bc301f1e71936230aa123f87f85a45c3f11bb3"} Jul 17 21:32:26.421273 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:26.421250 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7" Jul 17 21:32:26.513358 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:26.513320 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37a059fa-56e6-4f2e-98e2-1c79dfa639a2-util\") pod \"37a059fa-56e6-4f2e-98e2-1c79dfa639a2\" (UID: \"37a059fa-56e6-4f2e-98e2-1c79dfa639a2\") " Jul 17 21:32:26.513553 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:26.513374 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37a059fa-56e6-4f2e-98e2-1c79dfa639a2-bundle\") pod \"37a059fa-56e6-4f2e-98e2-1c79dfa639a2\" (UID: \"37a059fa-56e6-4f2e-98e2-1c79dfa639a2\") " Jul 17 21:32:26.513553 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:26.513439 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dlg8\" (UniqueName: \"kubernetes.io/projected/37a059fa-56e6-4f2e-98e2-1c79dfa639a2-kube-api-access-9dlg8\") pod \"37a059fa-56e6-4f2e-98e2-1c79dfa639a2\" (UID: \"37a059fa-56e6-4f2e-98e2-1c79dfa639a2\") " Jul 17 21:32:26.514145 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:26.514121 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a059fa-56e6-4f2e-98e2-1c79dfa639a2-bundle" (OuterVolumeSpecName: "bundle") pod "37a059fa-56e6-4f2e-98e2-1c79dfa639a2" (UID: "37a059fa-56e6-4f2e-98e2-1c79dfa639a2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:32:26.515637 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:26.515614 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a059fa-56e6-4f2e-98e2-1c79dfa639a2-kube-api-access-9dlg8" (OuterVolumeSpecName: "kube-api-access-9dlg8") pod "37a059fa-56e6-4f2e-98e2-1c79dfa639a2" (UID: "37a059fa-56e6-4f2e-98e2-1c79dfa639a2"). InnerVolumeSpecName "kube-api-access-9dlg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:32:26.518967 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:26.518935 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a059fa-56e6-4f2e-98e2-1c79dfa639a2-util" (OuterVolumeSpecName: "util") pod "37a059fa-56e6-4f2e-98e2-1c79dfa639a2" (UID: "37a059fa-56e6-4f2e-98e2-1c79dfa639a2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:32:26.614750 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:26.614667 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37a059fa-56e6-4f2e-98e2-1c79dfa639a2-bundle\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:32:26.614750 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:26.614698 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dlg8\" (UniqueName: \"kubernetes.io/projected/37a059fa-56e6-4f2e-98e2-1c79dfa639a2-kube-api-access-9dlg8\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:32:26.614750 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:26.614708 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37a059fa-56e6-4f2e-98e2-1c79dfa639a2-util\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:32:27.037176 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:32:27.037149 2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a059fa_56e6_4f2e_98e2_1c79dfa639a2.slice\": RecentStats: unable to find data in memory cache]" Jul 17 21:32:27.311172 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:27.311091 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7" event={"ID":"37a059fa-56e6-4f2e-98e2-1c79dfa639a2","Type":"ContainerDied","Data":"af4c0825ec21a1b65a49214e9138c6821528e47740c2d0fe67188d7bf9907a3e"} Jul 17 21:32:27.311172 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:27.311128 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgnxd7" Jul 17 21:32:27.311172 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:27.311142 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af4c0825ec21a1b65a49214e9138c6821528e47740c2d0fe67188d7bf9907a3e" Jul 17 21:32:33.510607 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:33.510574 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:34.514611 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.514576 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf"] Jul 17 21:32:34.514979 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.514868 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37a059fa-56e6-4f2e-98e2-1c79dfa639a2" containerName="util" Jul 17 21:32:34.514979 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.514880 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="37a059fa-56e6-4f2e-98e2-1c79dfa639a2" containerName="util" Jul 17 21:32:34.514979 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.514893 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37a059fa-56e6-4f2e-98e2-1c79dfa639a2" containerName="pull" Jul 17 21:32:34.514979 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.514899 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="37a059fa-56e6-4f2e-98e2-1c79dfa639a2" containerName="pull" Jul 17 21:32:34.514979 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.514910 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37a059fa-56e6-4f2e-98e2-1c79dfa639a2" containerName="extract" Jul 17 21:32:34.514979 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.514915 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="37a059fa-56e6-4f2e-98e2-1c79dfa639a2" containerName="extract" Jul 17 21:32:34.514979 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.514976 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="37a059fa-56e6-4f2e-98e2-1c79dfa639a2" containerName="extract" Jul 17 21:32:34.521342 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.521321 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" Jul 17 21:32:34.524938 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.524914 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 17 21:32:34.524938 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.524932 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 17 21:32:34.525096 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.524953 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9ws4k\"" Jul 17 21:32:34.525096 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.524937 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 17 21:32:34.525096 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.524934 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 17 21:32:34.542456 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.542431 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf"] Jul 17 21:32:34.579675 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.579640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48e2e9b0-f33c-418d-a7ce-852d2b31d50a-manifests\") pod \"opendatahub-operator-controller-manager-964b768d-r6hsf\" (UID: \"48e2e9b0-f33c-418d-a7ce-852d2b31d50a\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" Jul 17 21:32:34.579675 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.579674 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48e2e9b0-f33c-418d-a7ce-852d2b31d50a-tmp\") pod \"opendatahub-operator-controller-manager-964b768d-r6hsf\" (UID: \"48e2e9b0-f33c-418d-a7ce-852d2b31d50a\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" Jul 17 21:32:34.579861 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.579770 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48e2e9b0-f33c-418d-a7ce-852d2b31d50a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-964b768d-r6hsf\" (UID: \"48e2e9b0-f33c-418d-a7ce-852d2b31d50a\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" Jul 17 21:32:34.579861 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.579799 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ql8g\" (UniqueName: \"kubernetes.io/projected/48e2e9b0-f33c-418d-a7ce-852d2b31d50a-kube-api-access-4ql8g\") pod \"opendatahub-operator-controller-manager-964b768d-r6hsf\" (UID: \"48e2e9b0-f33c-418d-a7ce-852d2b31d50a\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" Jul 17 21:32:34.579861 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.579827 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48e2e9b0-f33c-418d-a7ce-852d2b31d50a-webhook-cert\") pod \"opendatahub-operator-controller-manager-964b768d-r6hsf\" (UID: \"48e2e9b0-f33c-418d-a7ce-852d2b31d50a\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" Jul 17 21:32:34.680739 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.680703 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48e2e9b0-f33c-418d-a7ce-852d2b31d50a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-964b768d-r6hsf\" (UID: \"48e2e9b0-f33c-418d-a7ce-852d2b31d50a\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" Jul 17 21:32:34.680739 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.680737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ql8g\" (UniqueName: \"kubernetes.io/projected/48e2e9b0-f33c-418d-a7ce-852d2b31d50a-kube-api-access-4ql8g\") pod \"opendatahub-operator-controller-manager-964b768d-r6hsf\" (UID: \"48e2e9b0-f33c-418d-a7ce-852d2b31d50a\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" Jul 17 21:32:34.680953 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.680757 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48e2e9b0-f33c-418d-a7ce-852d2b31d50a-webhook-cert\") pod \"opendatahub-operator-controller-manager-964b768d-r6hsf\" (UID: \"48e2e9b0-f33c-418d-a7ce-852d2b31d50a\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" Jul 17 21:32:34.680953 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.680787 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48e2e9b0-f33c-418d-a7ce-852d2b31d50a-manifests\") pod \"opendatahub-operator-controller-manager-964b768d-r6hsf\" (UID: \"48e2e9b0-f33c-418d-a7ce-852d2b31d50a\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" Jul 17 21:32:34.680953 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.680809 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48e2e9b0-f33c-418d-a7ce-852d2b31d50a-tmp\") pod \"opendatahub-operator-controller-manager-964b768d-r6hsf\" (UID: \"48e2e9b0-f33c-418d-a7ce-852d2b31d50a\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" Jul 17 21:32:34.681226 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.681200 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48e2e9b0-f33c-418d-a7ce-852d2b31d50a-manifests\") pod \"opendatahub-operator-controller-manager-964b768d-r6hsf\" (UID: \"48e2e9b0-f33c-418d-a7ce-852d2b31d50a\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" Jul 17 21:32:34.683316 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.683289 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48e2e9b0-f33c-418d-a7ce-852d2b31d50a-tmp\") pod \"opendatahub-operator-controller-manager-964b768d-r6hsf\" (UID: \"48e2e9b0-f33c-418d-a7ce-852d2b31d50a\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" Jul 17 21:32:34.683470 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.683361 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48e2e9b0-f33c-418d-a7ce-852d2b31d50a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-964b768d-r6hsf\" (UID: \"48e2e9b0-f33c-418d-a7ce-852d2b31d50a\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" Jul 17 21:32:34.683530 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.683513 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48e2e9b0-f33c-418d-a7ce-852d2b31d50a-webhook-cert\") pod \"opendatahub-operator-controller-manager-964b768d-r6hsf\" (UID: \"48e2e9b0-f33c-418d-a7ce-852d2b31d50a\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" Jul 17 21:32:34.697072 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.697047 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ql8g\" (UniqueName: \"kubernetes.io/projected/48e2e9b0-f33c-418d-a7ce-852d2b31d50a-kube-api-access-4ql8g\") pod \"opendatahub-operator-controller-manager-964b768d-r6hsf\" (UID: \"48e2e9b0-f33c-418d-a7ce-852d2b31d50a\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" Jul 17 21:32:34.831489 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.831376 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" Jul 17 21:32:34.897426 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.897360 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:34.969366 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.969342 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf"] Jul 17 21:32:34.971968 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:32:34.971943 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e2e9b0_f33c_418d_a7ce_852d2b31d50a.slice/crio-e840c8aa2c20eb5c29873cb199b84f55d3d1480f7fdb3399042341778d31fd26 WatchSource:0}: Error finding container e840c8aa2c20eb5c29873cb199b84f55d3d1480f7fdb3399042341778d31fd26: Status 404 returned error can't find the container with id e840c8aa2c20eb5c29873cb199b84f55d3d1480f7fdb3399042341778d31fd26 Jul 17 21:32:34.974173 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:34.974143 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:35.040597 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:35.040569 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:35.140621 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:35.140543 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:35.241741 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:35.241707 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:35.345032 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:35.344992 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" event={"ID":"48e2e9b0-f33c-418d-a7ce-852d2b31d50a","Type":"ContainerStarted","Data":"e840c8aa2c20eb5c29873cb199b84f55d3d1480f7fdb3399042341778d31fd26"} Jul 17 21:32:37.344682 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:37.344646 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:37.435591 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:37.435553 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:38.360978 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:38.360933 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" event={"ID":"48e2e9b0-f33c-418d-a7ce-852d2b31d50a","Type":"ContainerStarted","Data":"fb735db1a8b07e70d67641035308c16622fee6bf56a91574c6961a9a8ed4d5c9"} Jul 17 21:32:38.910432 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:38.910382 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 21:32:38.911721 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:38.911699 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 21:32:38.913703 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:38.913684 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 17 21:32:39.366406 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:39.366358 2580 generic.go:356] "Generic (PLEG): container finished" podID="48e2e9b0-f33c-418d-a7ce-852d2b31d50a" containerID="fb735db1a8b07e70d67641035308c16622fee6bf56a91574c6961a9a8ed4d5c9" exitCode=0 Jul 17 21:32:39.368611 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:39.366450 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" event={"ID":"48e2e9b0-f33c-418d-a7ce-852d2b31d50a","Type":"ContainerDied","Data":"fb735db1a8b07e70d67641035308c16622fee6bf56a91574c6961a9a8ed4d5c9"} Jul 17 21:32:40.373544 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:40.373501 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" event={"ID":"48e2e9b0-f33c-418d-a7ce-852d2b31d50a","Type":"ContainerStarted","Data":"0dd4d7a6d2578c3c9fcc10963ad9b97ed8142628a48788cb9d04b23e7f961e7c"} Jul 17 21:32:40.374020 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:40.373617 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" Jul 17 21:32:40.395596 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:40.395539 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" podStartSLOduration=2.155452127 podStartE2EDuration="6.395520636s" totalImagesPullingTime="3.233729861s" totalInitContainerRuntime="1.006338648s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:32:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:32:40.395262068 +0000 UTC m=+301.816981331" watchObservedRunningTime="2026-07-17 21:32:40.395520636 +0000 UTC m=+301.817239871" Jul 17 21:32:51.380459 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:51.380432 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-r6hsf" Jul 17 21:32:53.700923 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:53.700882 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:53.795134 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:53.795095 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:54.067792 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.067760 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:54.159913 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.159868 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:54.196974 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.196942 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-788f6fc9d-h598z"] Jul 17 21:32:54.205129 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.205109 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-788f6fc9d-h598z" Jul 17 21:32:54.209270 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.209243 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 17 21:32:54.209415 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.209243 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 17 21:32:54.209779 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.209759 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 21:32:54.209880 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.209780 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 21:32:54.209880 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.209788 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8kwkm\"" Jul 17 21:32:54.216605 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.216586 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-788f6fc9d-h598z"] Jul 17 21:32:54.360447 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.360344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8f4fcdb-b37c-471e-89f6-850451fa8af4-tmp\") pod \"kube-auth-proxy-788f6fc9d-h598z\" (UID: \"f8f4fcdb-b37c-471e-89f6-850451fa8af4\") " pod="openshift-ingress/kube-auth-proxy-788f6fc9d-h598z" Jul 17 21:32:54.360447 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.360412 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8f4fcdb-b37c-471e-89f6-850451fa8af4-tls-certs\") pod \"kube-auth-proxy-788f6fc9d-h598z\" (UID: \"f8f4fcdb-b37c-471e-89f6-850451fa8af4\") " pod="openshift-ingress/kube-auth-proxy-788f6fc9d-h598z" Jul 17 21:32:54.360626 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.360464 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffztj\" (UniqueName: \"kubernetes.io/projected/f8f4fcdb-b37c-471e-89f6-850451fa8af4-kube-api-access-ffztj\") pod \"kube-auth-proxy-788f6fc9d-h598z\" (UID: \"f8f4fcdb-b37c-471e-89f6-850451fa8af4\") " pod="openshift-ingress/kube-auth-proxy-788f6fc9d-h598z" Jul 17 21:32:54.399167 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.399131 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-54bpn"] Jul 17 21:32:54.407175 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.407141 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-54bpn" Jul 17 21:32:54.410993 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.410967 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 17 21:32:54.411140 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.410977 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2mp6n\"" Jul 17 21:32:54.420501 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.420478 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-54bpn"] Jul 17 21:32:54.450846 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.450810 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:54.460897 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.460875 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8f4fcdb-b37c-471e-89f6-850451fa8af4-tls-certs\") pod \"kube-auth-proxy-788f6fc9d-h598z\" (UID: \"f8f4fcdb-b37c-471e-89f6-850451fa8af4\") " pod="openshift-ingress/kube-auth-proxy-788f6fc9d-h598z" Jul 17 21:32:54.461017 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.460927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffztj\" (UniqueName: \"kubernetes.io/projected/f8f4fcdb-b37c-471e-89f6-850451fa8af4-kube-api-access-ffztj\") pod \"kube-auth-proxy-788f6fc9d-h598z\" (UID: \"f8f4fcdb-b37c-471e-89f6-850451fa8af4\") " pod="openshift-ingress/kube-auth-proxy-788f6fc9d-h598z" Jul 17 21:32:54.461075 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.461030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8f4fcdb-b37c-471e-89f6-850451fa8af4-tmp\") pod \"kube-auth-proxy-788f6fc9d-h598z\" (UID: \"f8f4fcdb-b37c-471e-89f6-850451fa8af4\") " pod="openshift-ingress/kube-auth-proxy-788f6fc9d-h598z" Jul 17 21:32:54.463345 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.463322 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8f4fcdb-b37c-471e-89f6-850451fa8af4-tmp\") pod \"kube-auth-proxy-788f6fc9d-h598z\" (UID: \"f8f4fcdb-b37c-471e-89f6-850451fa8af4\") " pod="openshift-ingress/kube-auth-proxy-788f6fc9d-h598z" Jul 17 21:32:54.463480 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.463466 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8f4fcdb-b37c-471e-89f6-850451fa8af4-tls-certs\") pod \"kube-auth-proxy-788f6fc9d-h598z\" (UID: \"f8f4fcdb-b37c-471e-89f6-850451fa8af4\") " pod="openshift-ingress/kube-auth-proxy-788f6fc9d-h598z" Jul 17 21:32:54.470501 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.470476 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffztj\" (UniqueName: \"kubernetes.io/projected/f8f4fcdb-b37c-471e-89f6-850451fa8af4-kube-api-access-ffztj\") pod \"kube-auth-proxy-788f6fc9d-h598z\" (UID: \"f8f4fcdb-b37c-471e-89f6-850451fa8af4\") " pod="openshift-ingress/kube-auth-proxy-788f6fc9d-h598z" Jul 17 21:32:54.515147 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.515120 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-788f6fc9d-h598z" Jul 17 21:32:54.544694 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.544662 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:54.561947 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.561801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npdd4\" (UniqueName: \"kubernetes.io/projected/88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb-kube-api-access-npdd4\") pod \"odh-model-controller-7998c67dc4-54bpn\" (UID: \"88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb\") " pod="opendatahub/odh-model-controller-7998c67dc4-54bpn" Jul 17 21:32:54.561947 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.561841 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb-cert\") pod \"odh-model-controller-7998c67dc4-54bpn\" (UID: \"88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb\") " pod="opendatahub/odh-model-controller-7998c67dc4-54bpn" Jul 17 21:32:54.652850 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.652764 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:54.653191 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.653170 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-788f6fc9d-h598z"] Jul 17 21:32:54.656201 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:32:54.656168 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f4fcdb_b37c_471e_89f6_850451fa8af4.slice/crio-402fccd2b6b0c1fa1ee4a7441d68fa09261cabdb6d472fdbfec6dbca34c94888 WatchSource:0}: Error finding container 402fccd2b6b0c1fa1ee4a7441d68fa09261cabdb6d472fdbfec6dbca34c94888: Status 404 returned error can't find the container with id 402fccd2b6b0c1fa1ee4a7441d68fa09261cabdb6d472fdbfec6dbca34c94888 Jul 17 21:32:54.658546 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.658525 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 21:32:54.662967 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.662942 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npdd4\" (UniqueName: \"kubernetes.io/projected/88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb-kube-api-access-npdd4\") pod \"odh-model-controller-7998c67dc4-54bpn\" (UID: \"88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb\") " pod="opendatahub/odh-model-controller-7998c67dc4-54bpn" Jul 17 21:32:54.663032 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.662985 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb-cert\") pod \"odh-model-controller-7998c67dc4-54bpn\" (UID: \"88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb\") " pod="opendatahub/odh-model-controller-7998c67dc4-54bpn" Jul 17 21:32:54.663182 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:32:54.663163 2580 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 17 21:32:54.663246 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:32:54.663235 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb-cert podName:88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb nodeName:}" failed. No retries permitted until 2026-07-17 21:32:55.16321554 +0000 UTC m=+316.584934753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb-cert") pod "odh-model-controller-7998c67dc4-54bpn" (UID: "88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb") : secret "odh-model-controller-webhook-cert" not found Jul 17 21:32:54.674954 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.674923 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npdd4\" (UniqueName: \"kubernetes.io/projected/88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb-kube-api-access-npdd4\") pod \"odh-model-controller-7998c67dc4-54bpn\" (UID: \"88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb\") " pod="opendatahub/odh-model-controller-7998c67dc4-54bpn" Jul 17 21:32:54.789666 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.789629 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:54.975729 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:54.975641 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:55.167840 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.167809 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb-cert\") pod \"odh-model-controller-7998c67dc4-54bpn\" (UID: \"88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb\") " pod="opendatahub/odh-model-controller-7998c67dc4-54bpn" Jul 17 21:32:55.170421 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.170369 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb-cert\") pod \"odh-model-controller-7998c67dc4-54bpn\" (UID: \"88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb\") " pod="opendatahub/odh-model-controller-7998c67dc4-54bpn" Jul 17 21:32:55.185848 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.185824 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:55.277467 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.277433 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jwq2n"] Jul 17 21:32:55.282338 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.282315 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-jwq2n" Jul 17 21:32:55.285057 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.285035 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 17 21:32:55.285203 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.285061 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-nf4jn\"" Jul 17 21:32:55.292951 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.292929 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jwq2n"] Jul 17 21:32:55.320494 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.320467 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-54bpn" Jul 17 21:32:55.446912 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.446879 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-788f6fc9d-h598z" event={"ID":"f8f4fcdb-b37c-471e-89f6-850451fa8af4","Type":"ContainerStarted","Data":"402fccd2b6b0c1fa1ee4a7441d68fa09261cabdb6d472fdbfec6dbca34c94888"} Jul 17 21:32:55.470948 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.470911 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w86fb\" (UniqueName: \"kubernetes.io/projected/3f2fc54c-9516-4d2c-b5ac-0910dca90e4b-kube-api-access-w86fb\") pod \"model-serving-api-54d89bd5fc-jwq2n\" (UID: \"3f2fc54c-9516-4d2c-b5ac-0910dca90e4b\") " pod="opendatahub/model-serving-api-54d89bd5fc-jwq2n" Jul 17 21:32:55.471116 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.471001 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f2fc54c-9516-4d2c-b5ac-0910dca90e4b-tls-certs\") pod \"model-serving-api-54d89bd5fc-jwq2n\" (UID: \"3f2fc54c-9516-4d2c-b5ac-0910dca90e4b\") " pod="opendatahub/model-serving-api-54d89bd5fc-jwq2n" Jul 17 21:32:55.488183 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.488157 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-54bpn"] Jul 17 21:32:55.491374 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:32:55.491343 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d14f31_60b7_4d3c_bf6d_e4ff760d1dfb.slice/crio-7097441e6fa629ad3142525e4dd6a297aa0a15eb2dc3005532a9a2c90d97bb9a WatchSource:0}: Error finding container 7097441e6fa629ad3142525e4dd6a297aa0a15eb2dc3005532a9a2c90d97bb9a: Status 404 returned error can't find the container with id 7097441e6fa629ad3142525e4dd6a297aa0a15eb2dc3005532a9a2c90d97bb9a Jul 17 21:32:55.571946 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.571861 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w86fb\" (UniqueName: \"kubernetes.io/projected/3f2fc54c-9516-4d2c-b5ac-0910dca90e4b-kube-api-access-w86fb\") pod \"model-serving-api-54d89bd5fc-jwq2n\" (UID: \"3f2fc54c-9516-4d2c-b5ac-0910dca90e4b\") " pod="opendatahub/model-serving-api-54d89bd5fc-jwq2n" Jul 17 21:32:55.572121 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.571949 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f2fc54c-9516-4d2c-b5ac-0910dca90e4b-tls-certs\") pod \"model-serving-api-54d89bd5fc-jwq2n\" (UID: \"3f2fc54c-9516-4d2c-b5ac-0910dca90e4b\") " pod="opendatahub/model-serving-api-54d89bd5fc-jwq2n" Jul 17 21:32:55.575086 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.575060 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f2fc54c-9516-4d2c-b5ac-0910dca90e4b-tls-certs\") pod \"model-serving-api-54d89bd5fc-jwq2n\" (UID: \"3f2fc54c-9516-4d2c-b5ac-0910dca90e4b\") " pod="opendatahub/model-serving-api-54d89bd5fc-jwq2n" Jul 17 21:32:55.579528 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.579480 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:55.585897 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.585863 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w86fb\" (UniqueName: \"kubernetes.io/projected/3f2fc54c-9516-4d2c-b5ac-0910dca90e4b-kube-api-access-w86fb\") pod \"model-serving-api-54d89bd5fc-jwq2n\" (UID: \"3f2fc54c-9516-4d2c-b5ac-0910dca90e4b\") " pod="opendatahub/model-serving-api-54d89bd5fc-jwq2n" Jul 17 21:32:55.595598 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.595578 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-jwq2n" Jul 17 21:32:55.624923 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.624425 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn"] Jul 17 21:32:55.628330 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.628308 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.631681 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.631657 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 17 21:32:55.632226 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.631938 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 17 21:32:55.632226 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.632204 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 17 21:32:55.632725 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.632704 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 17 21:32:55.633071 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.632905 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kqn6x\"" Jul 17 21:32:55.642717 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.642487 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn"] Jul 17 21:32:55.667369 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.667272 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:32:55.760984 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.760958 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jwq2n"] Jul 17 21:32:55.762299 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:32:55.762264 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2fc54c_9516_4d2c_b5ac_0910dca90e4b.slice/crio-8a3803c87c8eb2b328394b711b5ae13ba037a58c91f2c2e2f53f779839560525 WatchSource:0}: Error finding container 8a3803c87c8eb2b328394b711b5ae13ba037a58c91f2c2e2f53f779839560525: Status 404 returned error can't find the container with id 8a3803c87c8eb2b328394b711b5ae13ba037a58c91f2c2e2f53f779839560525 Jul 17 21:32:55.774219 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.774188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/746e88c9-f675-4e62-811b-5cddd7a272c2-cacerts\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.774341 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.774252 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/746e88c9-f675-4e62-811b-5cddd7a272c2-local-certs\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.774341 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.774284 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.774341 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.774312 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.774511 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.774428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-token\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.774511 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.774481 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.774511 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.774507 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2jcm\" (UniqueName: \"kubernetes.io/projected/746e88c9-f675-4e62-811b-5cddd7a272c2-kube-api-access-h2jcm\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.875064 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.874957 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-token\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.875064 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.875023 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.875064 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.875050 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2jcm\" (UniqueName: \"kubernetes.io/projected/746e88c9-f675-4e62-811b-5cddd7a272c2-kube-api-access-h2jcm\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.875668 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.875131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/746e88c9-f675-4e62-811b-5cddd7a272c2-cacerts\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.875668 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.875162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/746e88c9-f675-4e62-811b-5cddd7a272c2-local-certs\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.875668 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.875196 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.875668 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.875229 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.876228 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.876121 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.878166 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.878067 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/746e88c9-f675-4e62-811b-5cddd7a272c2-local-certs\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.878368 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.878346 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/746e88c9-f675-4e62-811b-5cddd7a272c2-cacerts\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.878476 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.878350 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.878726 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.878705 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.883615 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.883582 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-token\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.884052 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.884026 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2jcm\" (UniqueName: \"kubernetes.io/projected/746e88c9-f675-4e62-811b-5cddd7a272c2-kube-api-access-h2jcm\") pod \"istiod-openshift-gateway-7574459656-8f9cn\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:55.943602 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:55.943570 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:32:56.096539 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:56.096501 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn"] Jul 17 21:32:56.099567 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:32:56.099529 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746e88c9_f675_4e62_811b_5cddd7a272c2.slice/crio-c0a5d08e2f1732038904d68a12822f4d9f5e75aed194daf206553c0101300131 WatchSource:0}: Error finding container c0a5d08e2f1732038904d68a12822f4d9f5e75aed194daf206553c0101300131: Status 404 returned error can't find the container with id c0a5d08e2f1732038904d68a12822f4d9f5e75aed194daf206553c0101300131 Jul 17 21:32:56.456260 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:56.456220 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" event={"ID":"746e88c9-f675-4e62-811b-5cddd7a272c2","Type":"ContainerStarted","Data":"c0a5d08e2f1732038904d68a12822f4d9f5e75aed194daf206553c0101300131"} Jul 17 21:32:56.458044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:56.458011 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-jwq2n" event={"ID":"3f2fc54c-9516-4d2c-b5ac-0910dca90e4b","Type":"ContainerStarted","Data":"8a3803c87c8eb2b328394b711b5ae13ba037a58c91f2c2e2f53f779839560525"} Jul 17 21:32:56.459856 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:56.459827 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-54bpn" event={"ID":"88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb","Type":"ContainerStarted","Data":"7097441e6fa629ad3142525e4dd6a297aa0a15eb2dc3005532a9a2c90d97bb9a"} Jul 17 21:32:56.588840 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:32:56.588778 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:33:00.472124 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:00.472081 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:33:00.472449 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:00.472165 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:33:00.484101 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:00.484055 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-788f6fc9d-h598z" event={"ID":"f8f4fcdb-b37c-471e-89f6-850451fa8af4","Type":"ContainerStarted","Data":"6f66f1e2701860852615b424337887a0a375a320d304192bf447af947fa101ef"} Jul 17 21:33:00.503209 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:00.503167 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-788f6fc9d-h598z" podStartSLOduration=1.666511942 podStartE2EDuration="6.503151379s" totalImagesPullingTime="4.836639437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:33:00.502305119 +0000 UTC m=+321.924024368" watchObservedRunningTime="2026-07-17 21:33:00.503151379 +0000 UTC m=+321.924870611" Jul 17 21:33:00.977363 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:00.977316 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:33:01.078104 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:01.078055 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:33:01.291188 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:01.291140 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:33:01.404573 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:01.404535 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:33:01.499410 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:01.499347 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" event={"ID":"746e88c9-f675-4e62-811b-5cddd7a272c2","Type":"ContainerStarted","Data":"6c81de2c9a19bc43ee77ad982edff8db9b7dcf8e24db125dacc2ea6c978413fb"} Jul 17 21:33:01.499855 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:01.499473 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:33:01.501497 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:01.501469 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-jwq2n" event={"ID":"3f2fc54c-9516-4d2c-b5ac-0910dca90e4b","Type":"ContainerStarted","Data":"a6ce33f4001407a56d0beb5d40014a371000ce293f804aedaaac026cb08e8673"} Jul 17 21:33:01.501680 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:01.501623 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-jwq2n" Jul 17 21:33:01.501868 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:01.501848 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:33:01.503198 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:01.503178 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-54bpn" event={"ID":"88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb","Type":"ContainerStarted","Data":"cee6eb4c618e3f60409f7dfabd052150d016ec90bbbce6ff26f1478af7548800"} Jul 17 21:33:01.503412 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:01.503376 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-54bpn" Jul 17 21:33:01.530246 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:01.530195 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" podStartSLOduration=2.160354857 podStartE2EDuration="6.530181005s" totalImagesPullingTime="4.369826148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:33:01.528134119 +0000 UTC m=+322.949853350" watchObservedRunningTime="2026-07-17 21:33:01.530181005 +0000 UTC m=+322.951900215" Jul 17 21:33:01.571813 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:01.571734 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:33:01.572528 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:01.572483 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-jwq2n" podStartSLOduration=1.93181173 podStartE2EDuration="6.57246869s" totalImagesPullingTime="4.64065696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:33:01.570353977 +0000 UTC m=+322.992073235" watchObservedRunningTime="2026-07-17 21:33:01.57246869 +0000 UTC m=+322.994187922" Jul 17 21:33:01.589302 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:01.589258 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-54bpn" podStartSLOduration=2.6661672149999998 podStartE2EDuration="7.589241471s" totalImagesPullingTime="4.923074256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:33:01.588302458 +0000 UTC m=+323.010021710" watchObservedRunningTime="2026-07-17 21:33:01.589241471 +0000 UTC m=+323.010960703" Jul 17 21:33:04.107820 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.107785 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k"] Jul 17 21:33:04.111137 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.111122 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k" Jul 17 21:33:04.114480 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.114457 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 17 21:33:04.114606 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.114461 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 17 21:33:04.114806 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.114790 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 17 21:33:04.114856 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.114812 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 17 21:33:04.114894 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.114860 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 17 21:33:04.114940 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.114929 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5q5j2\"" Jul 17 21:33:04.120216 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.120195 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k"] Jul 17 21:33:04.156374 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.156344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qtcb\" (UniqueName: \"kubernetes.io/projected/2068a574-e2a0-4711-832b-7c51d9d2dabf-kube-api-access-2qtcb\") pod \"lws-controller-manager-7bbc9cc6cd-49q4k\" (UID: \"2068a574-e2a0-4711-832b-7c51d9d2dabf\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k" Jul 17 21:33:04.156581 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.156419 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2068a574-e2a0-4711-832b-7c51d9d2dabf-manager-config\") pod \"lws-controller-manager-7bbc9cc6cd-49q4k\" (UID: \"2068a574-e2a0-4711-832b-7c51d9d2dabf\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k" Jul 17 21:33:04.156581 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.156514 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2068a574-e2a0-4711-832b-7c51d9d2dabf-cert\") pod \"lws-controller-manager-7bbc9cc6cd-49q4k\" (UID: \"2068a574-e2a0-4711-832b-7c51d9d2dabf\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k" Jul 17 21:33:04.156689 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.156595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2068a574-e2a0-4711-832b-7c51d9d2dabf-metrics-cert\") pod \"lws-controller-manager-7bbc9cc6cd-49q4k\" (UID: \"2068a574-e2a0-4711-832b-7c51d9d2dabf\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k" Jul 17 21:33:04.257550 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.257512 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2068a574-e2a0-4711-832b-7c51d9d2dabf-manager-config\") pod \"lws-controller-manager-7bbc9cc6cd-49q4k\" (UID: \"2068a574-e2a0-4711-832b-7c51d9d2dabf\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k" Jul 17 21:33:04.257722 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.257572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2068a574-e2a0-4711-832b-7c51d9d2dabf-cert\") pod \"lws-controller-manager-7bbc9cc6cd-49q4k\" (UID: \"2068a574-e2a0-4711-832b-7c51d9d2dabf\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k" Jul 17 21:33:04.257722 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.257634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2068a574-e2a0-4711-832b-7c51d9d2dabf-metrics-cert\") pod \"lws-controller-manager-7bbc9cc6cd-49q4k\" (UID: \"2068a574-e2a0-4711-832b-7c51d9d2dabf\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k" Jul 17 21:33:04.257722 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.257671 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qtcb\" (UniqueName: \"kubernetes.io/projected/2068a574-e2a0-4711-832b-7c51d9d2dabf-kube-api-access-2qtcb\") pod \"lws-controller-manager-7bbc9cc6cd-49q4k\" (UID: \"2068a574-e2a0-4711-832b-7c51d9d2dabf\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k" Jul 17 21:33:04.258307 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.258277 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2068a574-e2a0-4711-832b-7c51d9d2dabf-manager-config\") pod \"lws-controller-manager-7bbc9cc6cd-49q4k\" (UID: \"2068a574-e2a0-4711-832b-7c51d9d2dabf\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k" Jul 17 21:33:04.260163 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.260142 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2068a574-e2a0-4711-832b-7c51d9d2dabf-cert\") pod \"lws-controller-manager-7bbc9cc6cd-49q4k\" (UID: \"2068a574-e2a0-4711-832b-7c51d9d2dabf\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k" Jul 17 21:33:04.260260 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.260187 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2068a574-e2a0-4711-832b-7c51d9d2dabf-metrics-cert\") pod \"lws-controller-manager-7bbc9cc6cd-49q4k\" (UID: \"2068a574-e2a0-4711-832b-7c51d9d2dabf\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k" Jul 17 21:33:04.270349 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.270321 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qtcb\" (UniqueName: \"kubernetes.io/projected/2068a574-e2a0-4711-832b-7c51d9d2dabf-kube-api-access-2qtcb\") pod \"lws-controller-manager-7bbc9cc6cd-49q4k\" (UID: \"2068a574-e2a0-4711-832b-7c51d9d2dabf\") " pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k" Jul 17 21:33:04.421368 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.421268 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k" Jul 17 21:33:04.557729 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:04.557707 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k"] Jul 17 21:33:04.559820 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:33:04.559789 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2068a574_e2a0_4711_832b_7c51d9d2dabf.slice/crio-95768ac9f6a82ff07cff5b54b20fa4883b958c1dae2476b6149df6f75845ee31 WatchSource:0}: Error finding container 95768ac9f6a82ff07cff5b54b20fa4883b958c1dae2476b6149df6f75845ee31: Status 404 returned error can't find the container with id 95768ac9f6a82ff07cff5b54b20fa4883b958c1dae2476b6149df6f75845ee31 Jul 17 21:33:05.526810 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:05.526773 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k" event={"ID":"2068a574-e2a0-4711-832b-7c51d9d2dabf","Type":"ContainerStarted","Data":"95768ac9f6a82ff07cff5b54b20fa4883b958c1dae2476b6149df6f75845ee31"} Jul 17 21:33:05.882207 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:05.882120 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:33:05.967632 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:05.967605 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:33:06.051865 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:06.051840 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:33:06.174992 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:06.174908 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:33:06.532648 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:06.532613 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k" event={"ID":"2068a574-e2a0-4711-832b-7c51d9d2dabf","Type":"ContainerStarted","Data":"b63ce4bcf386aea39fb621bd1e5231f6e5faaefcecc2820835ab345003734f6f"} Jul 17 21:33:06.533001 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:06.532699 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k" Jul 17 21:33:06.550266 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:06.550204 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k" podStartSLOduration=1.062041543 podStartE2EDuration="2.550188671s" totalImagesPullingTime="1.488147128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:33:06.548625454 +0000 UTC m=+327.970344728" watchObservedRunningTime="2026-07-17 21:33:06.550188671 +0000 UTC m=+327.971907977" Jul 17 21:33:06.872309 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:06.872227 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:33:07.167016 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:07.166918 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:33:07.350038 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:07.350002 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:33:07.434937 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:07.434846 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:33:08.759127 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:08.759091 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn"] Jul 17 21:33:08.759570 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:08.759273 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" podUID="746e88c9-f675-4e62-811b-5cddd7a272c2" containerName="discovery" containerID="cri-o://6c81de2c9a19bc43ee77ad982edff8db9b7dcf8e24db125dacc2ea6c978413fb" gracePeriod=30 Jul 17 21:33:09.036799 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.036775 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:33:09.108611 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.108582 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-csr-dns-cert\") pod \"746e88c9-f675-4e62-811b-5cddd7a272c2\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " Jul 17 21:33:09.108766 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.108633 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-kubeconfig\") pod \"746e88c9-f675-4e62-811b-5cddd7a272c2\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " Jul 17 21:33:09.108766 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.108654 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-token\") pod \"746e88c9-f675-4e62-811b-5cddd7a272c2\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " Jul 17 21:33:09.108766 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.108692 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/746e88c9-f675-4e62-811b-5cddd7a272c2-local-certs\") pod \"746e88c9-f675-4e62-811b-5cddd7a272c2\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " Jul 17 21:33:09.108932 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.108788 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2jcm\" (UniqueName: \"kubernetes.io/projected/746e88c9-f675-4e62-811b-5cddd7a272c2-kube-api-access-h2jcm\") pod \"746e88c9-f675-4e62-811b-5cddd7a272c2\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " Jul 17 21:33:09.108932 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.108824 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-csr-ca-configmap\") pod \"746e88c9-f675-4e62-811b-5cddd7a272c2\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " Jul 17 21:33:09.108932 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.108875 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/746e88c9-f675-4e62-811b-5cddd7a272c2-cacerts\") pod \"746e88c9-f675-4e62-811b-5cddd7a272c2\" (UID: \"746e88c9-f675-4e62-811b-5cddd7a272c2\") " Jul 17 21:33:09.109274 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.109239 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "746e88c9-f675-4e62-811b-5cddd7a272c2" (UID: "746e88c9-f675-4e62-811b-5cddd7a272c2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:33:09.111171 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.111141 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "746e88c9-f675-4e62-811b-5cddd7a272c2" (UID: "746e88c9-f675-4e62-811b-5cddd7a272c2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:33:09.111265 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.111170 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-token" (OuterVolumeSpecName: "istio-token") pod "746e88c9-f675-4e62-811b-5cddd7a272c2" (UID: "746e88c9-f675-4e62-811b-5cddd7a272c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:33:09.111374 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.111353 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "746e88c9-f675-4e62-811b-5cddd7a272c2" (UID: "746e88c9-f675-4e62-811b-5cddd7a272c2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:33:09.111502 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.111484 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746e88c9-f675-4e62-811b-5cddd7a272c2-kube-api-access-h2jcm" (OuterVolumeSpecName: "kube-api-access-h2jcm") pod "746e88c9-f675-4e62-811b-5cddd7a272c2" (UID: "746e88c9-f675-4e62-811b-5cddd7a272c2"). InnerVolumeSpecName "kube-api-access-h2jcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:33:09.111783 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.111760 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746e88c9-f675-4e62-811b-5cddd7a272c2-local-certs" (OuterVolumeSpecName: "local-certs") pod "746e88c9-f675-4e62-811b-5cddd7a272c2" (UID: "746e88c9-f675-4e62-811b-5cddd7a272c2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:33:09.111783 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.111761 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/746e88c9-f675-4e62-811b-5cddd7a272c2-cacerts" (OuterVolumeSpecName: "cacerts") pod "746e88c9-f675-4e62-811b-5cddd7a272c2" (UID: "746e88c9-f675-4e62-811b-5cddd7a272c2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:33:09.209763 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.209726 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-csr-dns-cert\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:33:09.209763 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.209758 2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-kubeconfig\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:33:09.209763 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.209767 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-token\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:33:09.209763 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.209775 2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/746e88c9-f675-4e62-811b-5cddd7a272c2-local-certs\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:33:09.210018 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.209782 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2jcm\" (UniqueName: \"kubernetes.io/projected/746e88c9-f675-4e62-811b-5cddd7a272c2-kube-api-access-h2jcm\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:33:09.210018 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.209790 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/746e88c9-f675-4e62-811b-5cddd7a272c2-istio-csr-ca-configmap\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:33:09.210018 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.209798 2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/746e88c9-f675-4e62-811b-5cddd7a272c2-cacerts\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:33:09.546686 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.546648 2580 generic.go:356] "Generic (PLEG): container finished" podID="746e88c9-f675-4e62-811b-5cddd7a272c2" containerID="6c81de2c9a19bc43ee77ad982edff8db9b7dcf8e24db125dacc2ea6c978413fb" exitCode=0 Jul 17 21:33:09.546913 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.546697 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" Jul 17 21:33:09.546913 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.546701 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" event={"ID":"746e88c9-f675-4e62-811b-5cddd7a272c2","Type":"ContainerDied","Data":"6c81de2c9a19bc43ee77ad982edff8db9b7dcf8e24db125dacc2ea6c978413fb"} Jul 17 21:33:09.546913 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.546749 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn" event={"ID":"746e88c9-f675-4e62-811b-5cddd7a272c2","Type":"ContainerDied","Data":"c0a5d08e2f1732038904d68a12822f4d9f5e75aed194daf206553c0101300131"} Jul 17 21:33:09.546913 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.546775 2580 scope.go:117] "RemoveContainer" containerID="6c81de2c9a19bc43ee77ad982edff8db9b7dcf8e24db125dacc2ea6c978413fb" Jul 17 21:33:09.555740 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.555724 2580 scope.go:117] "RemoveContainer" containerID="6c81de2c9a19bc43ee77ad982edff8db9b7dcf8e24db125dacc2ea6c978413fb" Jul 17 21:33:09.556019 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:33:09.555996 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c81de2c9a19bc43ee77ad982edff8db9b7dcf8e24db125dacc2ea6c978413fb\": container with ID starting with 6c81de2c9a19bc43ee77ad982edff8db9b7dcf8e24db125dacc2ea6c978413fb not found: ID does not exist" containerID="6c81de2c9a19bc43ee77ad982edff8db9b7dcf8e24db125dacc2ea6c978413fb" Jul 17 21:33:09.556058 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.556029 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c81de2c9a19bc43ee77ad982edff8db9b7dcf8e24db125dacc2ea6c978413fb"} err="failed to get container status \"6c81de2c9a19bc43ee77ad982edff8db9b7dcf8e24db125dacc2ea6c978413fb\": rpc error: code = NotFound desc = could not find container \"6c81de2c9a19bc43ee77ad982edff8db9b7dcf8e24db125dacc2ea6c978413fb\": container with ID starting with 6c81de2c9a19bc43ee77ad982edff8db9b7dcf8e24db125dacc2ea6c978413fb not found: ID does not exist" Jul 17 21:33:09.567901 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.567871 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn"] Jul 17 21:33:09.572407 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:09.572369 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-8f9cn"] Jul 17 21:33:11.022973 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:11.022941 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="746e88c9-f675-4e62-811b-5cddd7a272c2" path="/var/lib/kubelet/pods/746e88c9-f675-4e62-811b-5cddd7a272c2/volumes" Jul 17 21:33:12.510044 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:12.510014 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-54bpn" Jul 17 21:33:12.511872 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:12.511841 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-jwq2n" Jul 17 21:33:17.539830 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:17.539789 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7bbc9cc6cd-49q4k" Jul 17 21:33:53.913058 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:53.912990 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:33:53.970335 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:53.970301 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:33:55.175637 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:55.175600 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:33:55.271972 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:55.271938 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:33:55.372635 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:33:55.372604 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:10.690141 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:10.689417 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl"] Jul 17 21:34:10.691361 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:10.691220 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="746e88c9-f675-4e62-811b-5cddd7a272c2" containerName="discovery" Jul 17 21:34:10.691361 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:10.691247 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="746e88c9-f675-4e62-811b-5cddd7a272c2" containerName="discovery" Jul 17 21:34:10.692038 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:10.691760 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="746e88c9-f675-4e62-811b-5cddd7a272c2" containerName="discovery" Jul 17 21:34:10.695739 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:10.695719 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl" Jul 17 21:34:10.698453 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:10.698431 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 21:34:10.698804 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:10.698780 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kf4g2\"" Jul 17 21:34:10.699458 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:10.699439 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 21:34:10.699560 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:10.699438 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl"] Jul 17 21:34:10.827846 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:10.827811 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6wjs\" (UniqueName: \"kubernetes.io/projected/6b06a62b-c6dd-4204-9984-180bb2320534-kube-api-access-p6wjs\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl\" (UID: \"6b06a62b-c6dd-4204-9984-180bb2320534\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl" Jul 17 21:34:10.827846 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:10.827850 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b06a62b-c6dd-4204-9984-180bb2320534-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl\" (UID: \"6b06a62b-c6dd-4204-9984-180bb2320534\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl" Jul 17 21:34:10.828070 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:10.827878 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b06a62b-c6dd-4204-9984-180bb2320534-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl\" (UID: \"6b06a62b-c6dd-4204-9984-180bb2320534\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl" Jul 17 21:34:10.928591 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:10.928557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6wjs\" (UniqueName: \"kubernetes.io/projected/6b06a62b-c6dd-4204-9984-180bb2320534-kube-api-access-p6wjs\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl\" (UID: \"6b06a62b-c6dd-4204-9984-180bb2320534\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl" Jul 17 21:34:10.928591 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:10.928598 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b06a62b-c6dd-4204-9984-180bb2320534-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl\" (UID: \"6b06a62b-c6dd-4204-9984-180bb2320534\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl" Jul 17 21:34:10.928845 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:10.928622 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b06a62b-c6dd-4204-9984-180bb2320534-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl\" (UID: \"6b06a62b-c6dd-4204-9984-180bb2320534\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl" Jul 17 21:34:10.928978 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:10.928925 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b06a62b-c6dd-4204-9984-180bb2320534-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl\" (UID: \"6b06a62b-c6dd-4204-9984-180bb2320534\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl" Jul 17 21:34:10.929045 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:10.929017 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b06a62b-c6dd-4204-9984-180bb2320534-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl\" (UID: \"6b06a62b-c6dd-4204-9984-180bb2320534\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl" Jul 17 21:34:10.937145 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:10.937121 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6wjs\" (UniqueName: \"kubernetes.io/projected/6b06a62b-c6dd-4204-9984-180bb2320534-kube-api-access-p6wjs\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl\" (UID: \"6b06a62b-c6dd-4204-9984-180bb2320534\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl" Jul 17 21:34:11.007200 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.007124 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl" Jul 17 21:34:11.144008 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.143982 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl"] Jul 17 21:34:11.146077 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:34:11.146045 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b06a62b_c6dd_4204_9984_180bb2320534.slice/crio-eb38f19773e2a56c1255ba524faa8b7379d049c5ce01bbd832f8bfb7f96822bf WatchSource:0}: Error finding container eb38f19773e2a56c1255ba524faa8b7379d049c5ce01bbd832f8bfb7f96822bf: Status 404 returned error can't find the container with id eb38f19773e2a56c1255ba524faa8b7379d049c5ce01bbd832f8bfb7f96822bf Jul 17 21:34:11.296598 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.296563 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9"] Jul 17 21:34:11.300045 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.300023 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9" Jul 17 21:34:11.313117 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.313086 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9"] Jul 17 21:34:11.434106 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.434077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c5048e7-d826-4922-b3d5-29129e8cc5fd-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9\" (UID: \"3c5048e7-d826-4922-b3d5-29129e8cc5fd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9" Jul 17 21:34:11.434250 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.434118 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c5048e7-d826-4922-b3d5-29129e8cc5fd-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9\" (UID: \"3c5048e7-d826-4922-b3d5-29129e8cc5fd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9" Jul 17 21:34:11.434250 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.434137 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4fml\" (UniqueName: \"kubernetes.io/projected/3c5048e7-d826-4922-b3d5-29129e8cc5fd-kube-api-access-q4fml\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9\" (UID: \"3c5048e7-d826-4922-b3d5-29129e8cc5fd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9" Jul 17 21:34:11.535336 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.535293 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c5048e7-d826-4922-b3d5-29129e8cc5fd-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9\" (UID: \"3c5048e7-d826-4922-b3d5-29129e8cc5fd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9" Jul 17 21:34:11.535336 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.535340 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c5048e7-d826-4922-b3d5-29129e8cc5fd-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9\" (UID: \"3c5048e7-d826-4922-b3d5-29129e8cc5fd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9" Jul 17 21:34:11.535574 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.535359 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4fml\" (UniqueName: \"kubernetes.io/projected/3c5048e7-d826-4922-b3d5-29129e8cc5fd-kube-api-access-q4fml\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9\" (UID: \"3c5048e7-d826-4922-b3d5-29129e8cc5fd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9" Jul 17 21:34:11.535806 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.535782 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c5048e7-d826-4922-b3d5-29129e8cc5fd-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9\" (UID: \"3c5048e7-d826-4922-b3d5-29129e8cc5fd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9" Jul 17 21:34:11.535843 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.535795 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c5048e7-d826-4922-b3d5-29129e8cc5fd-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9\" (UID: \"3c5048e7-d826-4922-b3d5-29129e8cc5fd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9" Jul 17 21:34:11.547116 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.547064 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4fml\" (UniqueName: \"kubernetes.io/projected/3c5048e7-d826-4922-b3d5-29129e8cc5fd-kube-api-access-q4fml\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9\" (UID: \"3c5048e7-d826-4922-b3d5-29129e8cc5fd\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9" Jul 17 21:34:11.631107 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.631070 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9" Jul 17 21:34:11.764901 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.764872 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9"] Jul 17 21:34:11.769339 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:34:11.769308 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5048e7_d826_4922_b3d5_29129e8cc5fd.slice/crio-923d9d615409888f5eb3f44e3a2694c4a9085320cc244085e9986bb0483199a5 WatchSource:0}: Error finding container 923d9d615409888f5eb3f44e3a2694c4a9085320cc244085e9986bb0483199a5: Status 404 returned error can't find the container with id 923d9d615409888f5eb3f44e3a2694c4a9085320cc244085e9986bb0483199a5 Jul 17 21:34:11.843809 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.843774 2580 generic.go:356] "Generic (PLEG): container finished" podID="6b06a62b-c6dd-4204-9984-180bb2320534" containerID="1efc65a1a25711ebafc61cf25a9d13285d96501429a46ca319ed05ee08b6e577" exitCode=0 Jul 17 21:34:11.843958 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.843864 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl" event={"ID":"6b06a62b-c6dd-4204-9984-180bb2320534","Type":"ContainerDied","Data":"1efc65a1a25711ebafc61cf25a9d13285d96501429a46ca319ed05ee08b6e577"} Jul 17 21:34:11.843958 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.843914 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl" event={"ID":"6b06a62b-c6dd-4204-9984-180bb2320534","Type":"ContainerStarted","Data":"eb38f19773e2a56c1255ba524faa8b7379d049c5ce01bbd832f8bfb7f96822bf"} Jul 17 21:34:11.845557 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.845537 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9" event={"ID":"3c5048e7-d826-4922-b3d5-29129e8cc5fd","Type":"ContainerStarted","Data":"318a53b9e1924f0e6033934876a18654501a2904930a333fee7bbe7f6952a639"} Jul 17 21:34:11.845663 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.845577 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9" event={"ID":"3c5048e7-d826-4922-b3d5-29129e8cc5fd","Type":"ContainerStarted","Data":"923d9d615409888f5eb3f44e3a2694c4a9085320cc244085e9986bb0483199a5"} Jul 17 21:34:11.894362 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.894331 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj"] Jul 17 21:34:11.897704 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.897688 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj" Jul 17 21:34:11.909357 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.909336 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj"] Jul 17 21:34:11.939005 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.938981 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b53e33e-b212-4d0e-a4f5-af90b6f854b1-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj\" (UID: \"2b53e33e-b212-4d0e-a4f5-af90b6f854b1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj" Jul 17 21:34:11.939203 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.939183 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b53e33e-b212-4d0e-a4f5-af90b6f854b1-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj\" (UID: \"2b53e33e-b212-4d0e-a4f5-af90b6f854b1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj" Jul 17 21:34:11.939279 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:11.939221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh599\" (UniqueName: \"kubernetes.io/projected/2b53e33e-b212-4d0e-a4f5-af90b6f854b1-kube-api-access-kh599\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj\" (UID: \"2b53e33e-b212-4d0e-a4f5-af90b6f854b1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj" Jul 17 21:34:12.040206 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.040167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b53e33e-b212-4d0e-a4f5-af90b6f854b1-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj\" (UID: \"2b53e33e-b212-4d0e-a4f5-af90b6f854b1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj" Jul 17 21:34:12.040206 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.040206 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh599\" (UniqueName: \"kubernetes.io/projected/2b53e33e-b212-4d0e-a4f5-af90b6f854b1-kube-api-access-kh599\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj\" (UID: \"2b53e33e-b212-4d0e-a4f5-af90b6f854b1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj" Jul 17 21:34:12.040433 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.040268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b53e33e-b212-4d0e-a4f5-af90b6f854b1-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj\" (UID: \"2b53e33e-b212-4d0e-a4f5-af90b6f854b1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj" Jul 17 21:34:12.040597 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.040579 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b53e33e-b212-4d0e-a4f5-af90b6f854b1-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj\" (UID: \"2b53e33e-b212-4d0e-a4f5-af90b6f854b1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj" Jul 17 21:34:12.040646 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.040629 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b53e33e-b212-4d0e-a4f5-af90b6f854b1-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj\" (UID: \"2b53e33e-b212-4d0e-a4f5-af90b6f854b1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj" Jul 17 21:34:12.049800 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.049746 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh599\" (UniqueName: \"kubernetes.io/projected/2b53e33e-b212-4d0e-a4f5-af90b6f854b1-kube-api-access-kh599\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj\" (UID: \"2b53e33e-b212-4d0e-a4f5-af90b6f854b1\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj" Jul 17 21:34:12.251463 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.251431 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj" Jul 17 21:34:12.389123 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.389097 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj"] Jul 17 21:34:12.391117 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:34:12.391093 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b53e33e_b212_4d0e_a4f5_af90b6f854b1.slice/crio-adecac62b567d516a0b824e8e84ded7c8cb17d49c4378151a06f6443b4d8782d WatchSource:0}: Error finding container adecac62b567d516a0b824e8e84ded7c8cb17d49c4378151a06f6443b4d8782d: Status 404 returned error can't find the container with id adecac62b567d516a0b824e8e84ded7c8cb17d49c4378151a06f6443b4d8782d Jul 17 21:34:12.495701 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.495674 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z"] Jul 17 21:34:12.498988 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.498969 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z" Jul 17 21:34:12.508188 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.508164 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z"] Jul 17 21:34:12.544224 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.544199 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/deb01aec-306e-4a20-bee6-172990a40348-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z\" (UID: \"deb01aec-306e-4a20-bee6-172990a40348\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z" Jul 17 21:34:12.544332 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.544251 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/deb01aec-306e-4a20-bee6-172990a40348-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z\" (UID: \"deb01aec-306e-4a20-bee6-172990a40348\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z" Jul 17 21:34:12.544332 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.544310 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dldj6\" (UniqueName: \"kubernetes.io/projected/deb01aec-306e-4a20-bee6-172990a40348-kube-api-access-dldj6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z\" (UID: \"deb01aec-306e-4a20-bee6-172990a40348\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z" Jul 17 21:34:12.645490 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.645413 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dldj6\" (UniqueName: \"kubernetes.io/projected/deb01aec-306e-4a20-bee6-172990a40348-kube-api-access-dldj6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z\" (UID: \"deb01aec-306e-4a20-bee6-172990a40348\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z" Jul 17 21:34:12.645634 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.645500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/deb01aec-306e-4a20-bee6-172990a40348-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z\" (UID: \"deb01aec-306e-4a20-bee6-172990a40348\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z" Jul 17 21:34:12.645634 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.645559 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/deb01aec-306e-4a20-bee6-172990a40348-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z\" (UID: \"deb01aec-306e-4a20-bee6-172990a40348\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z" Jul 17 21:34:12.645876 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.645856 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/deb01aec-306e-4a20-bee6-172990a40348-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z\" (UID: \"deb01aec-306e-4a20-bee6-172990a40348\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z" Jul 17 21:34:12.645911 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.645885 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/deb01aec-306e-4a20-bee6-172990a40348-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z\" (UID: \"deb01aec-306e-4a20-bee6-172990a40348\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z" Jul 17 21:34:12.658112 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.658095 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dldj6\" (UniqueName: \"kubernetes.io/projected/deb01aec-306e-4a20-bee6-172990a40348-kube-api-access-dldj6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z\" (UID: \"deb01aec-306e-4a20-bee6-172990a40348\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z" Jul 17 21:34:12.844829 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.844795 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z" Jul 17 21:34:12.857334 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.857292 2580 generic.go:356] "Generic (PLEG): container finished" podID="2b53e33e-b212-4d0e-a4f5-af90b6f854b1" containerID="d68b9976c9a4fbe4b22584c71987fb8610d0aea5426242909d7fec3e72eb9d4a" exitCode=0 Jul 17 21:34:12.857490 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.857378 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj" event={"ID":"2b53e33e-b212-4d0e-a4f5-af90b6f854b1","Type":"ContainerDied","Data":"d68b9976c9a4fbe4b22584c71987fb8610d0aea5426242909d7fec3e72eb9d4a"} Jul 17 21:34:12.857550 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.857492 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj" event={"ID":"2b53e33e-b212-4d0e-a4f5-af90b6f854b1","Type":"ContainerStarted","Data":"adecac62b567d516a0b824e8e84ded7c8cb17d49c4378151a06f6443b4d8782d"} Jul 17 21:34:12.859056 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.859031 2580 generic.go:356] "Generic (PLEG): container finished" podID="3c5048e7-d826-4922-b3d5-29129e8cc5fd" containerID="318a53b9e1924f0e6033934876a18654501a2904930a333fee7bbe7f6952a639" exitCode=0 Jul 17 21:34:12.859162 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.859111 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9" event={"ID":"3c5048e7-d826-4922-b3d5-29129e8cc5fd","Type":"ContainerDied","Data":"318a53b9e1924f0e6033934876a18654501a2904930a333fee7bbe7f6952a639"} Jul 17 21:34:12.861299 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.861276 2580 generic.go:356] "Generic (PLEG): container finished" podID="6b06a62b-c6dd-4204-9984-180bb2320534" containerID="7d92e632c2e5d3048fb1940d907652e528019adc1178a3cea3c7c16686a9611f" exitCode=0 Jul 17 21:34:12.861377 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.861315 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl" event={"ID":"6b06a62b-c6dd-4204-9984-180bb2320534","Type":"ContainerDied","Data":"7d92e632c2e5d3048fb1940d907652e528019adc1178a3cea3c7c16686a9611f"} Jul 17 21:34:12.988138 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:12.988060 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z"] Jul 17 21:34:12.990190 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:34:12.990157 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb01aec_306e_4a20_bee6_172990a40348.slice/crio-d74a7383547cca399e65daf77e73fc8be434062d7d89e6c88f74c1de7c80e58f WatchSource:0}: Error finding container d74a7383547cca399e65daf77e73fc8be434062d7d89e6c88f74c1de7c80e58f: Status 404 returned error can't find the container with id d74a7383547cca399e65daf77e73fc8be434062d7d89e6c88f74c1de7c80e58f Jul 17 21:34:13.611111 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:13.611073 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:13.713642 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:13.713609 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:13.813231 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:13.813195 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:13.868064 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:13.867995 2580 generic.go:356] "Generic (PLEG): container finished" podID="2b53e33e-b212-4d0e-a4f5-af90b6f854b1" containerID="2a2c0df21c2b395b9fdf8a164a8ae451fe52a220c0ba49b776d5f4bd3598a514" exitCode=0 Jul 17 21:34:13.868474 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:13.868064 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj" event={"ID":"2b53e33e-b212-4d0e-a4f5-af90b6f854b1","Type":"ContainerDied","Data":"2a2c0df21c2b395b9fdf8a164a8ae451fe52a220c0ba49b776d5f4bd3598a514"} Jul 17 21:34:13.870095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:13.870076 2580 generic.go:356] "Generic (PLEG): container finished" podID="3c5048e7-d826-4922-b3d5-29129e8cc5fd" containerID="ed3cf53f9fb037c74c0af224a877a48cbc493fd921646ba13aee6c8baaf87fb4" exitCode=0 Jul 17 21:34:13.870178 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:13.870154 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9" event={"ID":"3c5048e7-d826-4922-b3d5-29129e8cc5fd","Type":"ContainerDied","Data":"ed3cf53f9fb037c74c0af224a877a48cbc493fd921646ba13aee6c8baaf87fb4"} Jul 17 21:34:13.871910 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:13.871888 2580 generic.go:356] "Generic (PLEG): container finished" podID="deb01aec-306e-4a20-bee6-172990a40348" containerID="270f739aa7c411a399682ec1fe0306898a6ca07685ae65d254477dc9ea8e9fce" exitCode=0 Jul 17 21:34:13.872007 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:13.871966 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z" event={"ID":"deb01aec-306e-4a20-bee6-172990a40348","Type":"ContainerDied","Data":"270f739aa7c411a399682ec1fe0306898a6ca07685ae65d254477dc9ea8e9fce"} Jul 17 21:34:13.872068 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:13.872008 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z" event={"ID":"deb01aec-306e-4a20-bee6-172990a40348","Type":"ContainerStarted","Data":"d74a7383547cca399e65daf77e73fc8be434062d7d89e6c88f74c1de7c80e58f"} Jul 17 21:34:13.874124 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:13.874102 2580 generic.go:356] "Generic (PLEG): container finished" podID="6b06a62b-c6dd-4204-9984-180bb2320534" containerID="62ac119f8565ec3f4626a06032a455e358198b8c8c167b94237a8f1309df15a6" exitCode=0 Jul 17 21:34:13.874222 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:13.874153 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl" event={"ID":"6b06a62b-c6dd-4204-9984-180bb2320534","Type":"ContainerDied","Data":"62ac119f8565ec3f4626a06032a455e358198b8c8c167b94237a8f1309df15a6"} Jul 17 21:34:14.364018 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:14.363982 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:14.428783 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:14.428753 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:14.881024 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:14.880996 2580 generic.go:356] "Generic (PLEG): container finished" podID="2b53e33e-b212-4d0e-a4f5-af90b6f854b1" containerID="f0c5cd65c50c2ec7ca4d24a0e7b95224fab075629730d22a574e494a4eaad78b" exitCode=0 Jul 17 21:34:14.881374 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:14.881083 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj" event={"ID":"2b53e33e-b212-4d0e-a4f5-af90b6f854b1","Type":"ContainerDied","Data":"f0c5cd65c50c2ec7ca4d24a0e7b95224fab075629730d22a574e494a4eaad78b"} Jul 17 21:34:14.883230 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:14.883205 2580 generic.go:356] "Generic (PLEG): container finished" podID="3c5048e7-d826-4922-b3d5-29129e8cc5fd" containerID="53c40987b65a684207fabfa53f5d69d26f1c41114452046f0d9d5ad4af31b238" exitCode=0 Jul 17 21:34:14.883329 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:14.883288 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9" event={"ID":"3c5048e7-d826-4922-b3d5-29129e8cc5fd","Type":"ContainerDied","Data":"53c40987b65a684207fabfa53f5d69d26f1c41114452046f0d9d5ad4af31b238"} Jul 17 21:34:14.885053 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:14.885031 2580 generic.go:356] "Generic (PLEG): container finished" podID="deb01aec-306e-4a20-bee6-172990a40348" containerID="86ae7dbac5c700e4ae4583e8a0757fc05db4fce0f5ee243a75a1a1cd8b31263c" exitCode=0 Jul 17 21:34:14.885130 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:14.885106 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z" event={"ID":"deb01aec-306e-4a20-bee6-172990a40348","Type":"ContainerDied","Data":"86ae7dbac5c700e4ae4583e8a0757fc05db4fce0f5ee243a75a1a1cd8b31263c"} Jul 17 21:34:15.072043 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:15.072016 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl" Jul 17 21:34:15.166647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:15.166618 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b06a62b-c6dd-4204-9984-180bb2320534-bundle\") pod \"6b06a62b-c6dd-4204-9984-180bb2320534\" (UID: \"6b06a62b-c6dd-4204-9984-180bb2320534\") " Jul 17 21:34:15.166802 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:15.166682 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b06a62b-c6dd-4204-9984-180bb2320534-util\") pod \"6b06a62b-c6dd-4204-9984-180bb2320534\" (UID: \"6b06a62b-c6dd-4204-9984-180bb2320534\") " Jul 17 21:34:15.166802 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:15.166747 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6wjs\" (UniqueName: \"kubernetes.io/projected/6b06a62b-c6dd-4204-9984-180bb2320534-kube-api-access-p6wjs\") pod \"6b06a62b-c6dd-4204-9984-180bb2320534\" (UID: \"6b06a62b-c6dd-4204-9984-180bb2320534\") " Jul 17 21:34:15.167181 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:15.167157 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b06a62b-c6dd-4204-9984-180bb2320534-bundle" (OuterVolumeSpecName: "bundle") pod "6b06a62b-c6dd-4204-9984-180bb2320534" (UID: "6b06a62b-c6dd-4204-9984-180bb2320534"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:34:15.169104 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:15.169083 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b06a62b-c6dd-4204-9984-180bb2320534-kube-api-access-p6wjs" (OuterVolumeSpecName: "kube-api-access-p6wjs") pod "6b06a62b-c6dd-4204-9984-180bb2320534" (UID: "6b06a62b-c6dd-4204-9984-180bb2320534"). InnerVolumeSpecName "kube-api-access-p6wjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:34:15.172452 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:15.172430 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b06a62b-c6dd-4204-9984-180bb2320534-util" (OuterVolumeSpecName: "util") pod "6b06a62b-c6dd-4204-9984-180bb2320534" (UID: "6b06a62b-c6dd-4204-9984-180bb2320534"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:34:15.267465 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:15.267428 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b06a62b-c6dd-4204-9984-180bb2320534-util\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:34:15.267621 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:15.267481 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6wjs\" (UniqueName: \"kubernetes.io/projected/6b06a62b-c6dd-4204-9984-180bb2320534-kube-api-access-p6wjs\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:34:15.267621 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:15.267491 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b06a62b-c6dd-4204-9984-180bb2320534-bundle\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:34:15.891705 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:15.891671 2580 generic.go:356] "Generic (PLEG): container finished" podID="deb01aec-306e-4a20-bee6-172990a40348" containerID="000c0e38e8142c8dbca4e5866ba3486e86d27daad33b3578508836bf8ae9ce24" exitCode=0 Jul 17 21:34:15.892155 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:15.891750 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z" event={"ID":"deb01aec-306e-4a20-bee6-172990a40348","Type":"ContainerDied","Data":"000c0e38e8142c8dbca4e5866ba3486e86d27daad33b3578508836bf8ae9ce24"} Jul 17 21:34:15.893630 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:15.893610 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl" Jul 17 21:34:15.893769 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:15.893642 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl" event={"ID":"6b06a62b-c6dd-4204-9984-180bb2320534","Type":"ContainerDied","Data":"eb38f19773e2a56c1255ba524faa8b7379d049c5ce01bbd832f8bfb7f96822bf"} Jul 17 21:34:15.893769 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:15.893678 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb38f19773e2a56c1255ba524faa8b7379d049c5ce01bbd832f8bfb7f96822bf" Jul 17 21:34:16.052924 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.052902 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj" Jul 17 21:34:16.055506 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.055491 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9" Jul 17 21:34:16.176548 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.176470 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b53e33e-b212-4d0e-a4f5-af90b6f854b1-bundle\") pod \"2b53e33e-b212-4d0e-a4f5-af90b6f854b1\" (UID: \"2b53e33e-b212-4d0e-a4f5-af90b6f854b1\") " Jul 17 21:34:16.176548 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.176511 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b53e33e-b212-4d0e-a4f5-af90b6f854b1-util\") pod \"2b53e33e-b212-4d0e-a4f5-af90b6f854b1\" (UID: \"2b53e33e-b212-4d0e-a4f5-af90b6f854b1\") " Jul 17 21:34:16.176548 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.176532 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c5048e7-d826-4922-b3d5-29129e8cc5fd-bundle\") pod \"3c5048e7-d826-4922-b3d5-29129e8cc5fd\" (UID: \"3c5048e7-d826-4922-b3d5-29129e8cc5fd\") " Jul 17 21:34:16.176810 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.176568 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh599\" (UniqueName: \"kubernetes.io/projected/2b53e33e-b212-4d0e-a4f5-af90b6f854b1-kube-api-access-kh599\") pod \"2b53e33e-b212-4d0e-a4f5-af90b6f854b1\" (UID: \"2b53e33e-b212-4d0e-a4f5-af90b6f854b1\") " Jul 17 21:34:16.176810 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.176596 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c5048e7-d826-4922-b3d5-29129e8cc5fd-util\") pod \"3c5048e7-d826-4922-b3d5-29129e8cc5fd\" (UID: \"3c5048e7-d826-4922-b3d5-29129e8cc5fd\") " Jul 17 21:34:16.176810 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.176617 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4fml\" (UniqueName: \"kubernetes.io/projected/3c5048e7-d826-4922-b3d5-29129e8cc5fd-kube-api-access-q4fml\") pod \"3c5048e7-d826-4922-b3d5-29129e8cc5fd\" (UID: \"3c5048e7-d826-4922-b3d5-29129e8cc5fd\") " Jul 17 21:34:16.177354 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.177236 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5048e7-d826-4922-b3d5-29129e8cc5fd-bundle" (OuterVolumeSpecName: "bundle") pod "3c5048e7-d826-4922-b3d5-29129e8cc5fd" (UID: "3c5048e7-d826-4922-b3d5-29129e8cc5fd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:34:16.177354 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.177246 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b53e33e-b212-4d0e-a4f5-af90b6f854b1-bundle" (OuterVolumeSpecName: "bundle") pod "2b53e33e-b212-4d0e-a4f5-af90b6f854b1" (UID: "2b53e33e-b212-4d0e-a4f5-af90b6f854b1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:34:16.178867 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.178807 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b53e33e-b212-4d0e-a4f5-af90b6f854b1-kube-api-access-kh599" (OuterVolumeSpecName: "kube-api-access-kh599") pod "2b53e33e-b212-4d0e-a4f5-af90b6f854b1" (UID: "2b53e33e-b212-4d0e-a4f5-af90b6f854b1"). InnerVolumeSpecName "kube-api-access-kh599". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:34:16.178959 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.178906 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5048e7-d826-4922-b3d5-29129e8cc5fd-kube-api-access-q4fml" (OuterVolumeSpecName: "kube-api-access-q4fml") pod "3c5048e7-d826-4922-b3d5-29129e8cc5fd" (UID: "3c5048e7-d826-4922-b3d5-29129e8cc5fd"). InnerVolumeSpecName "kube-api-access-q4fml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:34:16.182316 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.182295 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b53e33e-b212-4d0e-a4f5-af90b6f854b1-util" (OuterVolumeSpecName: "util") pod "2b53e33e-b212-4d0e-a4f5-af90b6f854b1" (UID: "2b53e33e-b212-4d0e-a4f5-af90b6f854b1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:34:16.182450 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.182430 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5048e7-d826-4922-b3d5-29129e8cc5fd-util" (OuterVolumeSpecName: "util") pod "3c5048e7-d826-4922-b3d5-29129e8cc5fd" (UID: "3c5048e7-d826-4922-b3d5-29129e8cc5fd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:34:16.277650 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.277617 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4fml\" (UniqueName: \"kubernetes.io/projected/3c5048e7-d826-4922-b3d5-29129e8cc5fd-kube-api-access-q4fml\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:34:16.277650 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.277646 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b53e33e-b212-4d0e-a4f5-af90b6f854b1-bundle\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:34:16.277650 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.277657 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b53e33e-b212-4d0e-a4f5-af90b6f854b1-util\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:34:16.277853 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.277665 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c5048e7-d826-4922-b3d5-29129e8cc5fd-bundle\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:34:16.277853 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.277673 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh599\" (UniqueName: \"kubernetes.io/projected/2b53e33e-b212-4d0e-a4f5-af90b6f854b1-kube-api-access-kh599\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:34:16.277853 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.277682 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c5048e7-d826-4922-b3d5-29129e8cc5fd-util\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:34:16.900621 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.900592 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj" Jul 17 21:34:16.901039 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.900592 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj" event={"ID":"2b53e33e-b212-4d0e-a4f5-af90b6f854b1","Type":"ContainerDied","Data":"adecac62b567d516a0b824e8e84ded7c8cb17d49c4378151a06f6443b4d8782d"} Jul 17 21:34:16.901039 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.900717 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adecac62b567d516a0b824e8e84ded7c8cb17d49c4378151a06f6443b4d8782d" Jul 17 21:34:16.902583 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.902561 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9" Jul 17 21:34:16.902701 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.902583 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9" event={"ID":"3c5048e7-d826-4922-b3d5-29129e8cc5fd","Type":"ContainerDied","Data":"923d9d615409888f5eb3f44e3a2694c4a9085320cc244085e9986bb0483199a5"} Jul 17 21:34:16.902701 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:16.902622 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="923d9d615409888f5eb3f44e3a2694c4a9085320cc244085e9986bb0483199a5" Jul 17 21:34:17.021754 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:17.021728 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z" Jul 17 21:34:17.083991 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:17.083958 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/deb01aec-306e-4a20-bee6-172990a40348-bundle\") pod \"deb01aec-306e-4a20-bee6-172990a40348\" (UID: \"deb01aec-306e-4a20-bee6-172990a40348\") " Jul 17 21:34:17.083991 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:17.083993 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dldj6\" (UniqueName: \"kubernetes.io/projected/deb01aec-306e-4a20-bee6-172990a40348-kube-api-access-dldj6\") pod \"deb01aec-306e-4a20-bee6-172990a40348\" (UID: \"deb01aec-306e-4a20-bee6-172990a40348\") " Jul 17 21:34:17.084180 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:17.084051 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/deb01aec-306e-4a20-bee6-172990a40348-util\") pod \"deb01aec-306e-4a20-bee6-172990a40348\" (UID: \"deb01aec-306e-4a20-bee6-172990a40348\") " Jul 17 21:34:17.084524 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:17.084490 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb01aec-306e-4a20-bee6-172990a40348-bundle" (OuterVolumeSpecName: "bundle") pod "deb01aec-306e-4a20-bee6-172990a40348" (UID: "deb01aec-306e-4a20-bee6-172990a40348"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:34:17.086458 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:17.086429 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb01aec-306e-4a20-bee6-172990a40348-kube-api-access-dldj6" (OuterVolumeSpecName: "kube-api-access-dldj6") pod "deb01aec-306e-4a20-bee6-172990a40348" (UID: "deb01aec-306e-4a20-bee6-172990a40348"). InnerVolumeSpecName "kube-api-access-dldj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:34:17.089777 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:17.089749 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb01aec-306e-4a20-bee6-172990a40348-util" (OuterVolumeSpecName: "util") pod "deb01aec-306e-4a20-bee6-172990a40348" (UID: "deb01aec-306e-4a20-bee6-172990a40348"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:34:17.185238 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:17.185149 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/deb01aec-306e-4a20-bee6-172990a40348-bundle\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:34:17.185238 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:17.185179 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dldj6\" (UniqueName: \"kubernetes.io/projected/deb01aec-306e-4a20-bee6-172990a40348-kube-api-access-dldj6\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:34:17.185238 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:17.185189 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/deb01aec-306e-4a20-bee6-172990a40348-util\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:34:17.909048 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:17.909021 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z" Jul 17 21:34:17.909048 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:17.909029 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z" event={"ID":"deb01aec-306e-4a20-bee6-172990a40348","Type":"ContainerDied","Data":"d74a7383547cca399e65daf77e73fc8be434062d7d89e6c88f74c1de7c80e58f"} Jul 17 21:34:17.909507 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:17.909072 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d74a7383547cca399e65daf77e73fc8be434062d7d89e6c88f74c1de7c80e58f" Jul 17 21:34:20.114493 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:20.114462 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:23.511341 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:23.511308 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:24.228047 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:24.228009 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:24.671820 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:24.671788 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:24.772159 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:24.772128 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:24.872059 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:24.872025 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:24.984542 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:24.984464 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:28.540265 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:28.540231 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:29.376804 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:29.376770 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:31.030912 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:31.030872 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:31.506739 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:31.506633 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:31.817549 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:31.817518 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:34.377560 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:34.377525 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:35.933694 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:35.933662 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:36.036898 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.036860 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:36.684533 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684499 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-svhjt"] Jul 17 21:34:36.684833 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684821 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb01aec-306e-4a20-bee6-172990a40348" containerName="util" Jul 17 21:34:36.684891 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684835 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb01aec-306e-4a20-bee6-172990a40348" containerName="util" Jul 17 21:34:36.684891 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684847 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b53e33e-b212-4d0e-a4f5-af90b6f854b1" containerName="util" Jul 17 21:34:36.684891 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684852 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b53e33e-b212-4d0e-a4f5-af90b6f854b1" containerName="util" Jul 17 21:34:36.684891 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684860 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c5048e7-d826-4922-b3d5-29129e8cc5fd" containerName="pull" Jul 17 21:34:36.684891 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684865 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c5048e7-d826-4922-b3d5-29129e8cc5fd" containerName="pull" Jul 17 21:34:36.684891 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684873 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c5048e7-d826-4922-b3d5-29129e8cc5fd" containerName="util" Jul 17 21:34:36.684891 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684879 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c5048e7-d826-4922-b3d5-29129e8cc5fd" containerName="util" Jul 17 21:34:36.684891 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684886 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb01aec-306e-4a20-bee6-172990a40348" containerName="extract" Jul 17 21:34:36.684891 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684892 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb01aec-306e-4a20-bee6-172990a40348" containerName="extract" Jul 17 21:34:36.685179 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684899 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b06a62b-c6dd-4204-9984-180bb2320534" containerName="extract" Jul 17 21:34:36.685179 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684904 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b06a62b-c6dd-4204-9984-180bb2320534" containerName="extract" Jul 17 21:34:36.685179 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684911 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c5048e7-d826-4922-b3d5-29129e8cc5fd" containerName="extract" Jul 17 21:34:36.685179 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684916 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c5048e7-d826-4922-b3d5-29129e8cc5fd" containerName="extract" Jul 17 21:34:36.685179 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684925 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b06a62b-c6dd-4204-9984-180bb2320534" containerName="util" Jul 17 21:34:36.685179 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684931 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b06a62b-c6dd-4204-9984-180bb2320534" containerName="util" Jul 17 21:34:36.685179 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684937 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b53e33e-b212-4d0e-a4f5-af90b6f854b1" containerName="pull" Jul 17 21:34:36.685179 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684942 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b53e33e-b212-4d0e-a4f5-af90b6f854b1" containerName="pull" Jul 17 21:34:36.685179 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684947 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b53e33e-b212-4d0e-a4f5-af90b6f854b1" containerName="extract" Jul 17 21:34:36.685179 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684951 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b53e33e-b212-4d0e-a4f5-af90b6f854b1" containerName="extract" Jul 17 21:34:36.685179 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684959 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b06a62b-c6dd-4204-9984-180bb2320534" containerName="pull" Jul 17 21:34:36.685179 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684963 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b06a62b-c6dd-4204-9984-180bb2320534" containerName="pull" Jul 17 21:34:36.685179 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684968 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb01aec-306e-4a20-bee6-172990a40348" containerName="pull" Jul 17 21:34:36.685179 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.684972 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb01aec-306e-4a20-bee6-172990a40348" containerName="pull" Jul 17 21:34:36.685179 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.685017 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb01aec-306e-4a20-bee6-172990a40348" containerName="extract" Jul 17 21:34:36.685179 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.685025 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c5048e7-d826-4922-b3d5-29129e8cc5fd" containerName="extract" Jul 17 21:34:36.685179 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.685031 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b53e33e-b212-4d0e-a4f5-af90b6f854b1" containerName="extract" Jul 17 21:34:36.685179 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.685037 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b06a62b-c6dd-4204-9984-180bb2320534" containerName="extract" Jul 17 21:34:36.689270 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.689254 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svhjt" Jul 17 21:34:36.692215 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.692192 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 21:34:36.693471 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.693185 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 21:34:36.693471 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.693211 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 17 21:34:36.693654 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.693525 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 17 21:34:36.693810 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.693794 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kf4g2\"" Jul 17 21:34:36.695942 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.695923 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-svhjt"] Jul 17 21:34:36.697752 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.697724 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:36.863746 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.863705 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt2gz\" (UniqueName: \"kubernetes.io/projected/c58d0a39-8de8-460b-83fb-6474831ee5b0-kube-api-access-zt2gz\") pod \"kuadrant-console-plugin-66b5774985-svhjt\" (UID: \"c58d0a39-8de8-460b-83fb-6474831ee5b0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svhjt" Jul 17 21:34:36.863926 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.863825 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c58d0a39-8de8-460b-83fb-6474831ee5b0-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-svhjt\" (UID: \"c58d0a39-8de8-460b-83fb-6474831ee5b0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svhjt" Jul 17 21:34:36.863926 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.863855 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c58d0a39-8de8-460b-83fb-6474831ee5b0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-svhjt\" (UID: \"c58d0a39-8de8-460b-83fb-6474831ee5b0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svhjt" Jul 17 21:34:36.964732 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.964624 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c58d0a39-8de8-460b-83fb-6474831ee5b0-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-svhjt\" (UID: \"c58d0a39-8de8-460b-83fb-6474831ee5b0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svhjt" Jul 17 21:34:36.964732 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.964681 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c58d0a39-8de8-460b-83fb-6474831ee5b0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-svhjt\" (UID: \"c58d0a39-8de8-460b-83fb-6474831ee5b0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svhjt" Jul 17 21:34:36.964732 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.964726 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt2gz\" (UniqueName: \"kubernetes.io/projected/c58d0a39-8de8-460b-83fb-6474831ee5b0-kube-api-access-zt2gz\") pod \"kuadrant-console-plugin-66b5774985-svhjt\" (UID: \"c58d0a39-8de8-460b-83fb-6474831ee5b0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svhjt" Jul 17 21:34:36.965272 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:34:36.964863 2580 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 17 21:34:36.965272 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:34:36.964952 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58d0a39-8de8-460b-83fb-6474831ee5b0-plugin-serving-cert podName:c58d0a39-8de8-460b-83fb-6474831ee5b0 nodeName:}" failed. No retries permitted until 2026-07-17 21:34:37.464929882 +0000 UTC m=+418.886649092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/c58d0a39-8de8-460b-83fb-6474831ee5b0-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-svhjt" (UID: "c58d0a39-8de8-460b-83fb-6474831ee5b0") : secret "plugin-serving-cert" not found Jul 17 21:34:36.965272 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.965234 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c58d0a39-8de8-460b-83fb-6474831ee5b0-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-svhjt\" (UID: \"c58d0a39-8de8-460b-83fb-6474831ee5b0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svhjt" Jul 17 21:34:36.977738 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:36.977713 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt2gz\" (UniqueName: \"kubernetes.io/projected/c58d0a39-8de8-460b-83fb-6474831ee5b0-kube-api-access-zt2gz\") pod \"kuadrant-console-plugin-66b5774985-svhjt\" (UID: \"c58d0a39-8de8-460b-83fb-6474831ee5b0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svhjt" Jul 17 21:34:37.037445 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:37.037410 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:37.469863 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:37.469817 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c58d0a39-8de8-460b-83fb-6474831ee5b0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-svhjt\" (UID: \"c58d0a39-8de8-460b-83fb-6474831ee5b0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svhjt" Jul 17 21:34:37.472534 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:37.472501 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c58d0a39-8de8-460b-83fb-6474831ee5b0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-svhjt\" (UID: \"c58d0a39-8de8-460b-83fb-6474831ee5b0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svhjt" Jul 17 21:34:37.600812 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:37.600778 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svhjt" Jul 17 21:34:37.735540 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:37.735509 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-svhjt"] Jul 17 21:34:37.738051 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:34:37.738015 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc58d0a39_8de8_460b_83fb_6474831ee5b0.slice/crio-95e0476226d5c114eda773e845d4ac3c9e3840cd7ac02dbcf160eec956e5a552 WatchSource:0}: Error finding container 95e0476226d5c114eda773e845d4ac3c9e3840cd7ac02dbcf160eec956e5a552: Status 404 returned error can't find the container with id 95e0476226d5c114eda773e845d4ac3c9e3840cd7ac02dbcf160eec956e5a552 Jul 17 21:34:38.011107 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:38.011076 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svhjt" event={"ID":"c58d0a39-8de8-460b-83fb-6474831ee5b0","Type":"ContainerStarted","Data":"95e0476226d5c114eda773e845d4ac3c9e3840cd7ac02dbcf160eec956e5a552"} Jul 17 21:34:39.323468 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:39.323409 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:39.423780 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:39.423704 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:45.675588 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:45.675544 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:45.768927 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:45.768881 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:47.461946 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:47.461911 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs"] Jul 17 21:34:47.495088 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:47.495052 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs"] Jul 17 21:34:47.495305 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:47.495172 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs" Jul 17 21:34:47.498907 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:47.498875 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zhf9w\"" Jul 17 21:34:47.562894 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:47.562849 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb673eda-70e8-4d86-b5c6-25ef35a99e4d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vf5xs\" (UID: \"fb673eda-70e8-4d86-b5c6-25ef35a99e4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs" Jul 17 21:34:47.563107 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:47.562973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k678\" (UniqueName: \"kubernetes.io/projected/fb673eda-70e8-4d86-b5c6-25ef35a99e4d-kube-api-access-6k678\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vf5xs\" (UID: \"fb673eda-70e8-4d86-b5c6-25ef35a99e4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs" Jul 17 21:34:47.664090 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:47.664051 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k678\" (UniqueName: \"kubernetes.io/projected/fb673eda-70e8-4d86-b5c6-25ef35a99e4d-kube-api-access-6k678\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vf5xs\" (UID: \"fb673eda-70e8-4d86-b5c6-25ef35a99e4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs" Jul 17 21:34:47.664279 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:47.664131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb673eda-70e8-4d86-b5c6-25ef35a99e4d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vf5xs\" (UID: \"fb673eda-70e8-4d86-b5c6-25ef35a99e4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs" Jul 17 21:34:47.664518 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:47.664492 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb673eda-70e8-4d86-b5c6-25ef35a99e4d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vf5xs\" (UID: \"fb673eda-70e8-4d86-b5c6-25ef35a99e4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs" Jul 17 21:34:47.672644 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:47.672615 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k678\" (UniqueName: \"kubernetes.io/projected/fb673eda-70e8-4d86-b5c6-25ef35a99e4d-kube-api-access-6k678\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vf5xs\" (UID: \"fb673eda-70e8-4d86-b5c6-25ef35a99e4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs" Jul 17 21:34:47.809772 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:47.809732 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs" Jul 17 21:34:52.142082 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:52.142030 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:52.239638 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:52.239594 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:54.823713 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:54.823678 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:54.925660 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:54.925628 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:57.161223 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:57.161156 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:57.229616 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:57.229582 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:34:58.895220 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:58.895193 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs"] Jul 17 21:34:58.896829 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:34:58.896805 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb673eda_70e8_4d86_b5c6_25ef35a99e4d.slice/crio-a9a1dbeee0e87601dee905955172ed68886501fa5861a6fb7cc3e32868b5808d WatchSource:0}: Error finding container a9a1dbeee0e87601dee905955172ed68886501fa5861a6fb7cc3e32868b5808d: Status 404 returned error can't find the container with id a9a1dbeee0e87601dee905955172ed68886501fa5861a6fb7cc3e32868b5808d Jul 17 21:34:59.129782 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:59.129681 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs" event={"ID":"fb673eda-70e8-4d86-b5c6-25ef35a99e4d","Type":"ContainerStarted","Data":"a9a1dbeee0e87601dee905955172ed68886501fa5861a6fb7cc3e32868b5808d"} Jul 17 21:34:59.131293 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:59.131262 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svhjt" event={"ID":"c58d0a39-8de8-460b-83fb-6474831ee5b0","Type":"ContainerStarted","Data":"77ea336e51a6c93dad8149dbcb8d9eb6f6cd91a72e09d455bb43131722ec30a2"} Jul 17 21:34:59.149843 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:34:59.149790 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svhjt" podStartSLOduration=2.039752875 podStartE2EDuration="23.14977149s" totalImagesPullingTime="21.110018615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:34:59.147660745 +0000 UTC m=+440.569379980" watchObservedRunningTime="2026-07-17 21:34:59.14977149 +0000 UTC m=+440.571490723" Jul 17 21:35:03.154249 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:03.154138 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs" event={"ID":"fb673eda-70e8-4d86-b5c6-25ef35a99e4d","Type":"ContainerStarted","Data":"9ea226843eee6c59e45dc994b5e828611f925e0759d1e1fb0ccc30d551bc6ec9"} Jul 17 21:35:03.154249 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:03.154208 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs" Jul 17 21:35:03.176306 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:03.176249 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs" podStartSLOduration=12.266373893 podStartE2EDuration="16.176231844s" totalImagesPullingTime="3.909857951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:35:03.173879821 +0000 UTC m=+444.595599068" watchObservedRunningTime="2026-07-17 21:35:03.176231844 +0000 UTC m=+444.597951076" Jul 17 21:35:04.269224 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:04.269189 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs"] Jul 17 21:35:05.141079 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:05.141013 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:05.164344 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:05.164288 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs" podUID="fb673eda-70e8-4d86-b5c6-25ef35a99e4d" containerName="manager" containerID="cri-o://9ea226843eee6c59e45dc994b5e828611f925e0759d1e1fb0ccc30d551bc6ec9" gracePeriod=10 Jul 17 21:35:05.247595 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:05.247561 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:05.454587 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:05.454551 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs" Jul 17 21:35:05.536921 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:05.536885 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k678\" (UniqueName: \"kubernetes.io/projected/fb673eda-70e8-4d86-b5c6-25ef35a99e4d-kube-api-access-6k678\") pod \"fb673eda-70e8-4d86-b5c6-25ef35a99e4d\" (UID: \"fb673eda-70e8-4d86-b5c6-25ef35a99e4d\") " Jul 17 21:35:05.537107 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:05.536978 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb673eda-70e8-4d86-b5c6-25ef35a99e4d-extensions-socket-volume\") pod \"fb673eda-70e8-4d86-b5c6-25ef35a99e4d\" (UID: \"fb673eda-70e8-4d86-b5c6-25ef35a99e4d\") " Jul 17 21:35:05.537533 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:05.537503 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb673eda-70e8-4d86-b5c6-25ef35a99e4d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fb673eda-70e8-4d86-b5c6-25ef35a99e4d" (UID: "fb673eda-70e8-4d86-b5c6-25ef35a99e4d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:35:05.539106 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:05.539068 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb673eda-70e8-4d86-b5c6-25ef35a99e4d-kube-api-access-6k678" (OuterVolumeSpecName: "kube-api-access-6k678") pod "fb673eda-70e8-4d86-b5c6-25ef35a99e4d" (UID: "fb673eda-70e8-4d86-b5c6-25ef35a99e4d"). InnerVolumeSpecName "kube-api-access-6k678". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:35:05.638300 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:05.638266 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k678\" (UniqueName: \"kubernetes.io/projected/fb673eda-70e8-4d86-b5c6-25ef35a99e4d-kube-api-access-6k678\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:35:05.638300 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:05.638295 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb673eda-70e8-4d86-b5c6-25ef35a99e4d-extensions-socket-volume\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:35:06.170899 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:06.170862 2580 generic.go:356] "Generic (PLEG): container finished" podID="fb673eda-70e8-4d86-b5c6-25ef35a99e4d" containerID="9ea226843eee6c59e45dc994b5e828611f925e0759d1e1fb0ccc30d551bc6ec9" exitCode=0 Jul 17 21:35:06.171081 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:06.170909 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs" event={"ID":"fb673eda-70e8-4d86-b5c6-25ef35a99e4d","Type":"ContainerDied","Data":"9ea226843eee6c59e45dc994b5e828611f925e0759d1e1fb0ccc30d551bc6ec9"} Jul 17 21:35:06.171081 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:06.170917 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs" Jul 17 21:35:06.171081 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:06.170953 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs" event={"ID":"fb673eda-70e8-4d86-b5c6-25ef35a99e4d","Type":"ContainerDied","Data":"a9a1dbeee0e87601dee905955172ed68886501fa5861a6fb7cc3e32868b5808d"} Jul 17 21:35:06.171081 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:06.170975 2580 scope.go:117] "RemoveContainer" containerID="9ea226843eee6c59e45dc994b5e828611f925e0759d1e1fb0ccc30d551bc6ec9" Jul 17 21:35:06.180193 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:06.180177 2580 scope.go:117] "RemoveContainer" containerID="9ea226843eee6c59e45dc994b5e828611f925e0759d1e1fb0ccc30d551bc6ec9" Jul 17 21:35:06.180502 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:35:06.180480 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea226843eee6c59e45dc994b5e828611f925e0759d1e1fb0ccc30d551bc6ec9\": container with ID starting with 9ea226843eee6c59e45dc994b5e828611f925e0759d1e1fb0ccc30d551bc6ec9 not found: ID does not exist" containerID="9ea226843eee6c59e45dc994b5e828611f925e0759d1e1fb0ccc30d551bc6ec9" Jul 17 21:35:06.180578 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:06.180514 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea226843eee6c59e45dc994b5e828611f925e0759d1e1fb0ccc30d551bc6ec9"} err="failed to get container status \"9ea226843eee6c59e45dc994b5e828611f925e0759d1e1fb0ccc30d551bc6ec9\": rpc error: code = NotFound desc = could not find container \"9ea226843eee6c59e45dc994b5e828611f925e0759d1e1fb0ccc30d551bc6ec9\": container with ID starting with 9ea226843eee6c59e45dc994b5e828611f925e0759d1e1fb0ccc30d551bc6ec9 not found: ID does not exist" Jul 17 21:35:06.195529 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:06.195502 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs"] Jul 17 21:35:06.202874 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:06.202850 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vf5xs"] Jul 17 21:35:07.024173 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:07.024138 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb673eda-70e8-4d86-b5c6-25ef35a99e4d" path="/var/lib/kubelet/pods/fb673eda-70e8-4d86-b5c6-25ef35a99e4d/volumes" Jul 17 21:35:11.567975 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:11.567937 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:11.667288 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:11.667252 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:12.144352 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:12.144317 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:12.285582 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:12.285536 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:12.873785 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:12.873752 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:20.631676 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:20.631639 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:21.119161 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:21.119121 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:22.360645 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:22.360611 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:22.463149 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:22.463100 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:22.534321 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:22.534267 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:23.146416 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.146361 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-x8qkb"] Jul 17 21:35:23.146774 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.146760 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb673eda-70e8-4d86-b5c6-25ef35a99e4d" containerName="manager" Jul 17 21:35:23.146832 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.146779 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb673eda-70e8-4d86-b5c6-25ef35a99e4d" containerName="manager" Jul 17 21:35:23.146870 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.146846 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb673eda-70e8-4d86-b5c6-25ef35a99e4d" containerName="manager" Jul 17 21:35:23.166139 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.166111 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-x8qkb"] Jul 17 21:35:23.166300 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.166201 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-x8qkb" Jul 17 21:35:23.169325 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.169299 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-g5bd7\"" Jul 17 21:35:23.177198 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.177172 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:23.277091 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.277034 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:23.293733 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.293703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sggjp\" (UniqueName: \"kubernetes.io/projected/0de6ce47-318a-46ec-8dec-da0c89bca30d-kube-api-access-sggjp\") pod \"authorino-58cd955675-x8qkb\" (UID: \"0de6ce47-318a-46ec-8dec-da0c89bca30d\") " pod="kuadrant-system/authorino-58cd955675-x8qkb" Jul 17 21:35:23.347701 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.347666 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-q2gjh"] Jul 17 21:35:23.350674 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.350658 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q2gjh" Jul 17 21:35:23.360590 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.360566 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q2gjh"] Jul 17 21:35:23.394827 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.394789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sggjp\" (UniqueName: \"kubernetes.io/projected/0de6ce47-318a-46ec-8dec-da0c89bca30d-kube-api-access-sggjp\") pod \"authorino-58cd955675-x8qkb\" (UID: \"0de6ce47-318a-46ec-8dec-da0c89bca30d\") " pod="kuadrant-system/authorino-58cd955675-x8qkb" Jul 17 21:35:23.402982 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.402908 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sggjp\" (UniqueName: \"kubernetes.io/projected/0de6ce47-318a-46ec-8dec-da0c89bca30d-kube-api-access-sggjp\") pod \"authorino-58cd955675-x8qkb\" (UID: \"0de6ce47-318a-46ec-8dec-da0c89bca30d\") " pod="kuadrant-system/authorino-58cd955675-x8qkb" Jul 17 21:35:23.474874 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.474834 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-x8qkb" Jul 17 21:35:23.495648 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.495598 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt5kd\" (UniqueName: \"kubernetes.io/projected/2e1c5b03-eec2-4567-95f8-c86936a05646-kube-api-access-mt5kd\") pod \"authorino-75d556bd77-q2gjh\" (UID: \"2e1c5b03-eec2-4567-95f8-c86936a05646\") " pod="kuadrant-system/authorino-75d556bd77-q2gjh" Jul 17 21:35:23.597006 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.596972 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt5kd\" (UniqueName: \"kubernetes.io/projected/2e1c5b03-eec2-4567-95f8-c86936a05646-kube-api-access-mt5kd\") pod \"authorino-75d556bd77-q2gjh\" (UID: \"2e1c5b03-eec2-4567-95f8-c86936a05646\") " pod="kuadrant-system/authorino-75d556bd77-q2gjh" Jul 17 21:35:23.604778 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.604755 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-x8qkb"] Jul 17 21:35:23.607010 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:35:23.606985 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0de6ce47_318a_46ec_8dec_da0c89bca30d.slice/crio-327a8307ce95b7e7ef10f65a999a02d0b7ed8299de4e5534b657137a45e7720e WatchSource:0}: Error finding container 327a8307ce95b7e7ef10f65a999a02d0b7ed8299de4e5534b657137a45e7720e: Status 404 returned error can't find the container with id 327a8307ce95b7e7ef10f65a999a02d0b7ed8299de4e5534b657137a45e7720e Jul 17 21:35:23.607208 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.607190 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt5kd\" (UniqueName: \"kubernetes.io/projected/2e1c5b03-eec2-4567-95f8-c86936a05646-kube-api-access-mt5kd\") pod \"authorino-75d556bd77-q2gjh\" (UID: \"2e1c5b03-eec2-4567-95f8-c86936a05646\") " pod="kuadrant-system/authorino-75d556bd77-q2gjh" Jul 17 21:35:23.659735 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.659653 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q2gjh" Jul 17 21:35:23.796217 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:23.796189 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q2gjh"] Jul 17 21:35:23.798001 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:35:23.797975 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e1c5b03_eec2_4567_95f8_c86936a05646.slice/crio-eab1f90d831eb805486e89e8cf72999519e64a2a1bc60c5882b6326caeca0ea9 WatchSource:0}: Error finding container eab1f90d831eb805486e89e8cf72999519e64a2a1bc60c5882b6326caeca0ea9: Status 404 returned error can't find the container with id eab1f90d831eb805486e89e8cf72999519e64a2a1bc60c5882b6326caeca0ea9 Jul 17 21:35:24.267790 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:24.267753 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-x8qkb" event={"ID":"0de6ce47-318a-46ec-8dec-da0c89bca30d","Type":"ContainerStarted","Data":"327a8307ce95b7e7ef10f65a999a02d0b7ed8299de4e5534b657137a45e7720e"} Jul 17 21:35:24.268944 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:24.268917 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q2gjh" event={"ID":"2e1c5b03-eec2-4567-95f8-c86936a05646","Type":"ContainerStarted","Data":"eab1f90d831eb805486e89e8cf72999519e64a2a1bc60c5882b6326caeca0ea9"} Jul 17 21:35:28.300239 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:28.300202 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q2gjh" event={"ID":"2e1c5b03-eec2-4567-95f8-c86936a05646","Type":"ContainerStarted","Data":"fa9e3ba06ffb259b9d282ff3650ddae98ca75708a49c37182818d61ed89b8d08"} Jul 17 21:35:28.301677 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:28.301651 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-x8qkb" event={"ID":"0de6ce47-318a-46ec-8dec-da0c89bca30d","Type":"ContainerStarted","Data":"92ef32d5c638f8351641c315282de272f32a4278e2e17ce3676f017cdfa1a621"} Jul 17 21:35:28.316491 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:28.316441 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-q2gjh" podStartSLOduration=1.864526939 podStartE2EDuration="5.316424564s" totalImagesPullingTime="3.451897625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:35:28.315321794 +0000 UTC m=+469.737041029" watchObservedRunningTime="2026-07-17 21:35:28.316424564 +0000 UTC m=+469.738143777" Jul 17 21:35:28.330228 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:28.330174 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-x8qkb" podStartSLOduration=1.6989084540000001 podStartE2EDuration="5.330156457s" totalImagesPullingTime="3.631248003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:35:28.328360548 +0000 UTC m=+469.750079776" watchObservedRunningTime="2026-07-17 21:35:28.330156457 +0000 UTC m=+469.751875689" Jul 17 21:35:28.355502 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:28.355469 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-x8qkb"] Jul 17 21:35:30.311881 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:30.311842 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-x8qkb" podUID="0de6ce47-318a-46ec-8dec-da0c89bca30d" containerName="authorino" containerID="cri-o://92ef32d5c638f8351641c315282de272f32a4278e2e17ce3676f017cdfa1a621" gracePeriod=30 Jul 17 21:35:30.569863 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:30.569795 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-x8qkb" Jul 17 21:35:30.653776 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:30.653746 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sggjp\" (UniqueName: \"kubernetes.io/projected/0de6ce47-318a-46ec-8dec-da0c89bca30d-kube-api-access-sggjp\") pod \"0de6ce47-318a-46ec-8dec-da0c89bca30d\" (UID: \"0de6ce47-318a-46ec-8dec-da0c89bca30d\") " Jul 17 21:35:30.656019 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:30.655989 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de6ce47-318a-46ec-8dec-da0c89bca30d-kube-api-access-sggjp" (OuterVolumeSpecName: "kube-api-access-sggjp") pod "0de6ce47-318a-46ec-8dec-da0c89bca30d" (UID: "0de6ce47-318a-46ec-8dec-da0c89bca30d"). InnerVolumeSpecName "kube-api-access-sggjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:35:30.754503 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:30.754467 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sggjp\" (UniqueName: \"kubernetes.io/projected/0de6ce47-318a-46ec-8dec-da0c89bca30d-kube-api-access-sggjp\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:35:31.318917 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:31.318826 2580 generic.go:356] "Generic (PLEG): container finished" podID="0de6ce47-318a-46ec-8dec-da0c89bca30d" containerID="92ef32d5c638f8351641c315282de272f32a4278e2e17ce3676f017cdfa1a621" exitCode=0 Jul 17 21:35:31.318917 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:31.318872 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-x8qkb" Jul 17 21:35:31.318917 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:31.318900 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-x8qkb" event={"ID":"0de6ce47-318a-46ec-8dec-da0c89bca30d","Type":"ContainerDied","Data":"92ef32d5c638f8351641c315282de272f32a4278e2e17ce3676f017cdfa1a621"} Jul 17 21:35:31.319485 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:31.318939 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-x8qkb" event={"ID":"0de6ce47-318a-46ec-8dec-da0c89bca30d","Type":"ContainerDied","Data":"327a8307ce95b7e7ef10f65a999a02d0b7ed8299de4e5534b657137a45e7720e"} Jul 17 21:35:31.319485 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:31.318960 2580 scope.go:117] "RemoveContainer" containerID="92ef32d5c638f8351641c315282de272f32a4278e2e17ce3676f017cdfa1a621" Jul 17 21:35:31.329787 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:31.329762 2580 scope.go:117] "RemoveContainer" containerID="92ef32d5c638f8351641c315282de272f32a4278e2e17ce3676f017cdfa1a621" Jul 17 21:35:31.330340 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:35:31.330316 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ef32d5c638f8351641c315282de272f32a4278e2e17ce3676f017cdfa1a621\": container with ID starting with 92ef32d5c638f8351641c315282de272f32a4278e2e17ce3676f017cdfa1a621 not found: ID does not exist" containerID="92ef32d5c638f8351641c315282de272f32a4278e2e17ce3676f017cdfa1a621" Jul 17 21:35:31.330455 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:31.330350 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ef32d5c638f8351641c315282de272f32a4278e2e17ce3676f017cdfa1a621"} err="failed to get container status \"92ef32d5c638f8351641c315282de272f32a4278e2e17ce3676f017cdfa1a621\": rpc error: code = NotFound desc = could not find container \"92ef32d5c638f8351641c315282de272f32a4278e2e17ce3676f017cdfa1a621\": container with ID starting with 92ef32d5c638f8351641c315282de272f32a4278e2e17ce3676f017cdfa1a621 not found: ID does not exist" Jul 17 21:35:31.344758 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:31.344734 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-x8qkb"] Jul 17 21:35:31.350155 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:31.350130 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-x8qkb"] Jul 17 21:35:33.028155 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:33.028118 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0de6ce47-318a-46ec-8dec-da0c89bca30d" path="/var/lib/kubelet/pods/0de6ce47-318a-46ec-8dec-da0c89bca30d/volumes" Jul 17 21:35:38.473007 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:38.472972 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:38.594953 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:38.594915 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:39.016525 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:39.016482 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:42.574002 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:42.573969 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:47.306135 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:47.306098 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:47.452679 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:47.452641 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:51.247965 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.247928 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:51.484982 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.484948 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7jm4r"] Jul 17 21:35:51.485314 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.485302 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0de6ce47-318a-46ec-8dec-da0c89bca30d" containerName="authorino" Jul 17 21:35:51.485357 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.485317 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0de6ce47-318a-46ec-8dec-da0c89bca30d" containerName="authorino" Jul 17 21:35:51.485477 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.485410 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0de6ce47-318a-46ec-8dec-da0c89bca30d" containerName="authorino" Jul 17 21:35:51.488247 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.488225 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7jm4r" Jul 17 21:35:51.496155 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.496132 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7jm4r"] Jul 17 21:35:51.520384 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.520347 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcpc8\" (UniqueName: \"kubernetes.io/projected/3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875-kube-api-access-kcpc8\") pod \"authorino-5d4b67c446-7jm4r\" (UID: \"3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875\") " pod="kuadrant-system/authorino-5d4b67c446-7jm4r" Jul 17 21:35:51.522510 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.522487 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:51.620932 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.620889 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcpc8\" (UniqueName: \"kubernetes.io/projected/3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875-kube-api-access-kcpc8\") pod \"authorino-5d4b67c446-7jm4r\" (UID: \"3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875\") " pod="kuadrant-system/authorino-5d4b67c446-7jm4r" Jul 17 21:35:51.631614 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.631585 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcpc8\" (UniqueName: \"kubernetes.io/projected/3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875-kube-api-access-kcpc8\") pod \"authorino-5d4b67c446-7jm4r\" (UID: \"3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875\") " pod="kuadrant-system/authorino-5d4b67c446-7jm4r" Jul 17 21:35:51.722175 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.722139 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7jm4r"] Jul 17 21:35:51.722361 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.722348 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7jm4r" Jul 17 21:35:51.750681 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.750651 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-758499b9f5-6v9t2"] Jul 17 21:35:51.752881 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.752865 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-758499b9f5-6v9t2" Jul 17 21:35:51.761486 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.761458 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-758499b9f5-6v9t2"] Jul 17 21:35:51.823155 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.823125 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9sgq\" (UniqueName: \"kubernetes.io/projected/fbb0364a-7c12-484e-b2b5-0d46a84d0213-kube-api-access-c9sgq\") pod \"authorino-758499b9f5-6v9t2\" (UID: \"fbb0364a-7c12-484e-b2b5-0d46a84d0213\") " pod="kuadrant-system/authorino-758499b9f5-6v9t2" Jul 17 21:35:51.862209 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.862182 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-758499b9f5-6v9t2"] Jul 17 21:35:51.862571 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:35:51.862546 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-c9sgq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-758499b9f5-6v9t2" podUID="fbb0364a-7c12-484e-b2b5-0d46a84d0213" Jul 17 21:35:51.864284 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:35:51.864256 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3dd901c9_2ad3_4f3f_bd6b_daa9e9bd8875.slice/crio-4d0b41f42bba12852c0aa4aa2ee48d69aaadf91bc36d54289cfe6493e815521c WatchSource:0}: Error finding container 4d0b41f42bba12852c0aa4aa2ee48d69aaadf91bc36d54289cfe6493e815521c: Status 404 returned error can't find the container with id 4d0b41f42bba12852c0aa4aa2ee48d69aaadf91bc36d54289cfe6493e815521c Jul 17 21:35:51.865105 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.865082 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7jm4r"] Jul 17 21:35:51.887066 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.887031 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c885449c9-j8jgr"] Jul 17 21:35:51.889302 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.889282 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c885449c9-j8jgr" Jul 17 21:35:51.891947 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.891926 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 17 21:35:51.897719 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.897696 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c885449c9-j8jgr"] Jul 17 21:35:51.923790 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.923755 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1462236-b596-4a88-b3c1-9fa2ad76f382-tls-cert\") pod \"authorino-c885449c9-j8jgr\" (UID: \"a1462236-b596-4a88-b3c1-9fa2ad76f382\") " pod="kuadrant-system/authorino-c885449c9-j8jgr" Jul 17 21:35:51.923978 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.923824 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brb4z\" (UniqueName: \"kubernetes.io/projected/a1462236-b596-4a88-b3c1-9fa2ad76f382-kube-api-access-brb4z\") pod \"authorino-c885449c9-j8jgr\" (UID: \"a1462236-b596-4a88-b3c1-9fa2ad76f382\") " pod="kuadrant-system/authorino-c885449c9-j8jgr" Jul 17 21:35:51.923978 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.923890 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9sgq\" (UniqueName: \"kubernetes.io/projected/fbb0364a-7c12-484e-b2b5-0d46a84d0213-kube-api-access-c9sgq\") pod \"authorino-758499b9f5-6v9t2\" (UID: \"fbb0364a-7c12-484e-b2b5-0d46a84d0213\") " pod="kuadrant-system/authorino-758499b9f5-6v9t2" Jul 17 21:35:51.932131 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:51.932109 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9sgq\" (UniqueName: \"kubernetes.io/projected/fbb0364a-7c12-484e-b2b5-0d46a84d0213-kube-api-access-c9sgq\") pod \"authorino-758499b9f5-6v9t2\" (UID: \"fbb0364a-7c12-484e-b2b5-0d46a84d0213\") " pod="kuadrant-system/authorino-758499b9f5-6v9t2" Jul 17 21:35:52.024902 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:52.024863 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brb4z\" (UniqueName: \"kubernetes.io/projected/a1462236-b596-4a88-b3c1-9fa2ad76f382-kube-api-access-brb4z\") pod \"authorino-c885449c9-j8jgr\" (UID: \"a1462236-b596-4a88-b3c1-9fa2ad76f382\") " pod="kuadrant-system/authorino-c885449c9-j8jgr" Jul 17 21:35:52.025083 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:52.024954 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1462236-b596-4a88-b3c1-9fa2ad76f382-tls-cert\") pod \"authorino-c885449c9-j8jgr\" (UID: \"a1462236-b596-4a88-b3c1-9fa2ad76f382\") " pod="kuadrant-system/authorino-c885449c9-j8jgr" Jul 17 21:35:52.027555 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:52.027492 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1462236-b596-4a88-b3c1-9fa2ad76f382-tls-cert\") pod \"authorino-c885449c9-j8jgr\" (UID: \"a1462236-b596-4a88-b3c1-9fa2ad76f382\") " pod="kuadrant-system/authorino-c885449c9-j8jgr" Jul 17 21:35:52.034718 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:52.034649 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brb4z\" (UniqueName: \"kubernetes.io/projected/a1462236-b596-4a88-b3c1-9fa2ad76f382-kube-api-access-brb4z\") pod \"authorino-c885449c9-j8jgr\" (UID: \"a1462236-b596-4a88-b3c1-9fa2ad76f382\") " pod="kuadrant-system/authorino-c885449c9-j8jgr" Jul 17 21:35:52.199970 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:52.199927 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c885449c9-j8jgr" Jul 17 21:35:52.331766 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:52.331744 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c885449c9-j8jgr"] Jul 17 21:35:52.333640 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:35:52.333610 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1462236_b596_4a88_b3c1_9fa2ad76f382.slice/crio-2730135de4519368b265524595b7749f5abbaf792e320b105c29c429c8700dcf WatchSource:0}: Error finding container 2730135de4519368b265524595b7749f5abbaf792e320b105c29c429c8700dcf: Status 404 returned error can't find the container with id 2730135de4519368b265524595b7749f5abbaf792e320b105c29c429c8700dcf Jul 17 21:35:52.429047 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:52.429012 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7jm4r" event={"ID":"3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875","Type":"ContainerStarted","Data":"4d0b41f42bba12852c0aa4aa2ee48d69aaadf91bc36d54289cfe6493e815521c"} Jul 17 21:35:52.430304 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:52.430284 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c885449c9-j8jgr" event={"ID":"a1462236-b596-4a88-b3c1-9fa2ad76f382","Type":"ContainerStarted","Data":"2730135de4519368b265524595b7749f5abbaf792e320b105c29c429c8700dcf"} Jul 17 21:35:52.430433 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:52.430305 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-758499b9f5-6v9t2" Jul 17 21:35:52.435945 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:52.435912 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-758499b9f5-6v9t2" Jul 17 21:35:52.528923 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:52.528886 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9sgq\" (UniqueName: \"kubernetes.io/projected/fbb0364a-7c12-484e-b2b5-0d46a84d0213-kube-api-access-c9sgq\") pod \"fbb0364a-7c12-484e-b2b5-0d46a84d0213\" (UID: \"fbb0364a-7c12-484e-b2b5-0d46a84d0213\") " Jul 17 21:35:52.531191 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:52.531158 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb0364a-7c12-484e-b2b5-0d46a84d0213-kube-api-access-c9sgq" (OuterVolumeSpecName: "kube-api-access-c9sgq") pod "fbb0364a-7c12-484e-b2b5-0d46a84d0213" (UID: "fbb0364a-7c12-484e-b2b5-0d46a84d0213"). InnerVolumeSpecName "kube-api-access-c9sgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:35:52.629724 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:52.629642 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9sgq\" (UniqueName: \"kubernetes.io/projected/fbb0364a-7c12-484e-b2b5-0d46a84d0213-kube-api-access-c9sgq\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:35:53.438133 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:53.438027 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c885449c9-j8jgr" event={"ID":"a1462236-b596-4a88-b3c1-9fa2ad76f382","Type":"ContainerStarted","Data":"e6dffc6b18ee6dbc572effe390e231108978cde2f07b3829616517a7ded33021"} Jul 17 21:35:53.442738 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:53.442711 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-758499b9f5-6v9t2" Jul 17 21:35:53.442911 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:53.442722 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-7jm4r" podUID="3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875" containerName="authorino" containerID="cri-o://c5f15ba67567a80397f3cf2edb7c249a1844351999148272bc802160f76cb23c" gracePeriod=30 Jul 17 21:35:53.442911 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:53.442744 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7jm4r" event={"ID":"3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875","Type":"ContainerStarted","Data":"c5f15ba67567a80397f3cf2edb7c249a1844351999148272bc802160f76cb23c"} Jul 17 21:35:53.456273 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:53.456103 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c885449c9-j8jgr" podStartSLOduration=2.093199824 podStartE2EDuration="2.456084602s" totalImagesPullingTime="362.884778ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:35:53.45216874 +0000 UTC m=+494.873887975" watchObservedRunningTime="2026-07-17 21:35:53.456084602 +0000 UTC m=+494.877803835" Jul 17 21:35:53.481066 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:53.479966 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q2gjh"] Jul 17 21:35:53.481066 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:53.480255 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-q2gjh" podUID="2e1c5b03-eec2-4567-95f8-c86936a05646" containerName="authorino" containerID="cri-o://fa9e3ba06ffb259b9d282ff3650ddae98ca75708a49c37182818d61ed89b8d08" gracePeriod=30 Jul 17 21:35:53.483779 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:53.483690 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-758499b9f5-6v9t2"] Jul 17 21:35:53.487045 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:53.487008 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-758499b9f5-6v9t2"] Jul 17 21:35:53.496255 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:53.496192 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-7jm4r" podStartSLOduration=1.988590772 podStartE2EDuration="2.496169012s" totalImagesPullingTime="507.57824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:35:53.496116636 +0000 UTC m=+494.917835870" watchObservedRunningTime="2026-07-17 21:35:53.496169012 +0000 UTC m=+494.917888246" Jul 17 21:35:53.751858 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:53.751834 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7jm4r" Jul 17 21:35:53.754715 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:53.754692 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q2gjh" Jul 17 21:35:53.840672 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:53.840632 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcpc8\" (UniqueName: \"kubernetes.io/projected/3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875-kube-api-access-kcpc8\") pod \"3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875\" (UID: \"3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875\") " Jul 17 21:35:53.840867 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:53.840700 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mt5kd\" (UniqueName: \"kubernetes.io/projected/2e1c5b03-eec2-4567-95f8-c86936a05646-kube-api-access-mt5kd\") pod \"2e1c5b03-eec2-4567-95f8-c86936a05646\" (UID: \"2e1c5b03-eec2-4567-95f8-c86936a05646\") " Jul 17 21:35:53.842958 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:53.842916 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875-kube-api-access-kcpc8" (OuterVolumeSpecName: "kube-api-access-kcpc8") pod "3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875" (UID: "3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875"). InnerVolumeSpecName "kube-api-access-kcpc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:35:53.843074 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:53.842960 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1c5b03-eec2-4567-95f8-c86936a05646-kube-api-access-mt5kd" (OuterVolumeSpecName: "kube-api-access-mt5kd") pod "2e1c5b03-eec2-4567-95f8-c86936a05646" (UID: "2e1c5b03-eec2-4567-95f8-c86936a05646"). InnerVolumeSpecName "kube-api-access-mt5kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:35:53.941481 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:53.941441 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcpc8\" (UniqueName: \"kubernetes.io/projected/3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875-kube-api-access-kcpc8\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:35:53.941481 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:53.941475 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt5kd\" (UniqueName: \"kubernetes.io/projected/2e1c5b03-eec2-4567-95f8-c86936a05646-kube-api-access-mt5kd\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:35:54.449114 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:54.449081 2580 generic.go:356] "Generic (PLEG): container finished" podID="3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875" containerID="c5f15ba67567a80397f3cf2edb7c249a1844351999148272bc802160f76cb23c" exitCode=0 Jul 17 21:35:54.449578 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:54.449131 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7jm4r" Jul 17 21:35:54.449578 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:54.449174 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7jm4r" event={"ID":"3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875","Type":"ContainerDied","Data":"c5f15ba67567a80397f3cf2edb7c249a1844351999148272bc802160f76cb23c"} Jul 17 21:35:54.449578 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:54.449218 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7jm4r" event={"ID":"3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875","Type":"ContainerDied","Data":"4d0b41f42bba12852c0aa4aa2ee48d69aaadf91bc36d54289cfe6493e815521c"} Jul 17 21:35:54.449578 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:54.449234 2580 scope.go:117] "RemoveContainer" containerID="c5f15ba67567a80397f3cf2edb7c249a1844351999148272bc802160f76cb23c" Jul 17 21:35:54.450660 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:54.450639 2580 generic.go:356] "Generic (PLEG): container finished" podID="2e1c5b03-eec2-4567-95f8-c86936a05646" containerID="fa9e3ba06ffb259b9d282ff3650ddae98ca75708a49c37182818d61ed89b8d08" exitCode=0 Jul 17 21:35:54.450741 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:54.450679 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q2gjh" Jul 17 21:35:54.450741 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:54.450726 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q2gjh" event={"ID":"2e1c5b03-eec2-4567-95f8-c86936a05646","Type":"ContainerDied","Data":"fa9e3ba06ffb259b9d282ff3650ddae98ca75708a49c37182818d61ed89b8d08"} Jul 17 21:35:54.450841 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:54.450803 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q2gjh" event={"ID":"2e1c5b03-eec2-4567-95f8-c86936a05646","Type":"ContainerDied","Data":"eab1f90d831eb805486e89e8cf72999519e64a2a1bc60c5882b6326caeca0ea9"} Jul 17 21:35:54.458275 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:54.458256 2580 scope.go:117] "RemoveContainer" containerID="c5f15ba67567a80397f3cf2edb7c249a1844351999148272bc802160f76cb23c" Jul 17 21:35:54.458636 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:35:54.458619 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5f15ba67567a80397f3cf2edb7c249a1844351999148272bc802160f76cb23c\": container with ID starting with c5f15ba67567a80397f3cf2edb7c249a1844351999148272bc802160f76cb23c not found: ID does not exist" containerID="c5f15ba67567a80397f3cf2edb7c249a1844351999148272bc802160f76cb23c" Jul 17 21:35:54.458726 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:54.458645 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f15ba67567a80397f3cf2edb7c249a1844351999148272bc802160f76cb23c"} err="failed to get container status \"c5f15ba67567a80397f3cf2edb7c249a1844351999148272bc802160f76cb23c\": rpc error: code = NotFound desc = could not find container \"c5f15ba67567a80397f3cf2edb7c249a1844351999148272bc802160f76cb23c\": container with ID starting with c5f15ba67567a80397f3cf2edb7c249a1844351999148272bc802160f76cb23c not found: ID does not exist" Jul 17 21:35:54.458726 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:54.458661 2580 scope.go:117] "RemoveContainer" containerID="fa9e3ba06ffb259b9d282ff3650ddae98ca75708a49c37182818d61ed89b8d08" Jul 17 21:35:54.466997 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:54.466977 2580 scope.go:117] "RemoveContainer" containerID="fa9e3ba06ffb259b9d282ff3650ddae98ca75708a49c37182818d61ed89b8d08" Jul 17 21:35:54.467281 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:35:54.467264 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa9e3ba06ffb259b9d282ff3650ddae98ca75708a49c37182818d61ed89b8d08\": container with ID starting with fa9e3ba06ffb259b9d282ff3650ddae98ca75708a49c37182818d61ed89b8d08 not found: ID does not exist" containerID="fa9e3ba06ffb259b9d282ff3650ddae98ca75708a49c37182818d61ed89b8d08" Jul 17 21:35:54.467319 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:54.467290 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa9e3ba06ffb259b9d282ff3650ddae98ca75708a49c37182818d61ed89b8d08"} err="failed to get container status \"fa9e3ba06ffb259b9d282ff3650ddae98ca75708a49c37182818d61ed89b8d08\": rpc error: code = NotFound desc = could not find container \"fa9e3ba06ffb259b9d282ff3650ddae98ca75708a49c37182818d61ed89b8d08\": container with ID starting with fa9e3ba06ffb259b9d282ff3650ddae98ca75708a49c37182818d61ed89b8d08 not found: ID does not exist" Jul 17 21:35:54.472743 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:54.472714 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q2gjh"] Jul 17 21:35:54.477340 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:54.477318 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q2gjh"] Jul 17 21:35:54.487562 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:54.487537 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7jm4r"] Jul 17 21:35:54.493909 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:54.493885 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7jm4r"] Jul 17 21:35:55.024293 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:55.024263 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e1c5b03-eec2-4567-95f8-c86936a05646" path="/var/lib/kubelet/pods/2e1c5b03-eec2-4567-95f8-c86936a05646/volumes" Jul 17 21:35:55.024609 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:55.024596 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875" path="/var/lib/kubelet/pods/3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875/volumes" Jul 17 21:35:55.024870 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:55.024859 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbb0364a-7c12-484e-b2b5-0d46a84d0213" path="/var/lib/kubelet/pods/fbb0364a-7c12-484e-b2b5-0d46a84d0213/volumes" Jul 17 21:35:57.290348 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:57.290316 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:58.347160 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.347122 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bf8dd86bf-v57mj"] Jul 17 21:35:58.347573 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.347481 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e1c5b03-eec2-4567-95f8-c86936a05646" containerName="authorino" Jul 17 21:35:58.347573 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.347492 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e1c5b03-eec2-4567-95f8-c86936a05646" containerName="authorino" Jul 17 21:35:58.347573 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.347504 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875" containerName="authorino" Jul 17 21:35:58.347573 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.347509 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875" containerName="authorino" Jul 17 21:35:58.347573 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.347572 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dd901c9-2ad3-4f3f-bd6b-daa9e9bd8875" containerName="authorino" Jul 17 21:35:58.347744 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.347584 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e1c5b03-eec2-4567-95f8-c86936a05646" containerName="authorino" Jul 17 21:35:58.351232 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.351198 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:58.358358 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.358335 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bf8dd86bf-v57mj" Jul 17 21:35:58.362517 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.362040 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 17 21:35:58.362517 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.362058 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-n7k8g\"" Jul 17 21:35:58.362517 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.362295 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 17 21:35:58.365385 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.365356 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bf8dd86bf-v57mj"] Jul 17 21:35:58.413256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.413221 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:58.483846 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.483816 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58g94\" (UniqueName: \"kubernetes.io/projected/9343c441-c85c-4538-b70c-f843620e9dd0-kube-api-access-58g94\") pod \"maas-controller-5bf8dd86bf-v57mj\" (UID: \"9343c441-c85c-4538-b70c-f843620e9dd0\") " pod="opendatahub/maas-controller-5bf8dd86bf-v57mj" Jul 17 21:35:58.484029 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.483876 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9343c441-c85c-4538-b70c-f843620e9dd0-webhook-cert\") pod \"maas-controller-5bf8dd86bf-v57mj\" (UID: \"9343c441-c85c-4538-b70c-f843620e9dd0\") " pod="opendatahub/maas-controller-5bf8dd86bf-v57mj" Jul 17 21:35:58.514836 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.514807 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:58.532160 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.532123 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86d8757495-9mqvf"] Jul 17 21:35:58.553115 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.553087 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86d8757495-9mqvf"] Jul 17 21:35:58.553291 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.553196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86d8757495-9mqvf" Jul 17 21:35:58.584780 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.584747 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58g94\" (UniqueName: \"kubernetes.io/projected/9343c441-c85c-4538-b70c-f843620e9dd0-kube-api-access-58g94\") pod \"maas-controller-5bf8dd86bf-v57mj\" (UID: \"9343c441-c85c-4538-b70c-f843620e9dd0\") " pod="opendatahub/maas-controller-5bf8dd86bf-v57mj" Jul 17 21:35:58.584912 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.584836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9343c441-c85c-4538-b70c-f843620e9dd0-webhook-cert\") pod \"maas-controller-5bf8dd86bf-v57mj\" (UID: \"9343c441-c85c-4538-b70c-f843620e9dd0\") " pod="opendatahub/maas-controller-5bf8dd86bf-v57mj" Jul 17 21:35:58.584963 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:35:58.584948 2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 17 21:35:58.585019 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:35:58.585008 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9343c441-c85c-4538-b70c-f843620e9dd0-webhook-cert podName:9343c441-c85c-4538-b70c-f843620e9dd0 nodeName:}" failed. No retries permitted until 2026-07-17 21:35:59.084991893 +0000 UTC m=+500.506711104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/9343c441-c85c-4538-b70c-f843620e9dd0-webhook-cert") pod "maas-controller-5bf8dd86bf-v57mj" (UID: "9343c441-c85c-4538-b70c-f843620e9dd0") : secret "maas-controller-webhook-cert" not found Jul 17 21:35:58.597003 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.596973 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58g94\" (UniqueName: \"kubernetes.io/projected/9343c441-c85c-4538-b70c-f843620e9dd0-kube-api-access-58g94\") pod \"maas-controller-5bf8dd86bf-v57mj\" (UID: \"9343c441-c85c-4538-b70c-f843620e9dd0\") " pod="opendatahub/maas-controller-5bf8dd86bf-v57mj" Jul 17 21:35:58.685628 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.685540 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a-webhook-cert\") pod \"maas-controller-86d8757495-9mqvf\" (UID: \"6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a\") " pod="opendatahub/maas-controller-86d8757495-9mqvf" Jul 17 21:35:58.685628 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.685618 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsvs5\" (UniqueName: \"kubernetes.io/projected/6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a-kube-api-access-zsvs5\") pod \"maas-controller-86d8757495-9mqvf\" (UID: \"6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a\") " pod="opendatahub/maas-controller-86d8757495-9mqvf" Jul 17 21:35:58.786673 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.786632 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsvs5\" (UniqueName: \"kubernetes.io/projected/6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a-kube-api-access-zsvs5\") pod \"maas-controller-86d8757495-9mqvf\" (UID: \"6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a\") " pod="opendatahub/maas-controller-86d8757495-9mqvf" Jul 17 21:35:58.786862 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.786726 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a-webhook-cert\") pod \"maas-controller-86d8757495-9mqvf\" (UID: \"6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a\") " pod="opendatahub/maas-controller-86d8757495-9mqvf" Jul 17 21:35:58.789459 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.789430 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a-webhook-cert\") pod \"maas-controller-86d8757495-9mqvf\" (UID: \"6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a\") " pod="opendatahub/maas-controller-86d8757495-9mqvf" Jul 17 21:35:58.794967 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.794939 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsvs5\" (UniqueName: \"kubernetes.io/projected/6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a-kube-api-access-zsvs5\") pod \"maas-controller-86d8757495-9mqvf\" (UID: \"6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a\") " pod="opendatahub/maas-controller-86d8757495-9mqvf" Jul 17 21:35:58.837099 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.837060 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:35:58.863237 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:58.863212 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86d8757495-9mqvf" Jul 17 21:35:59.017087 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:59.017057 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86d8757495-9mqvf"] Jul 17 21:35:59.018682 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:35:59.018651 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e46030f_efe6_4b4e_bd65_a4a3a11dcf0a.slice/crio-26d7cf6369c5e35d8b3193be45d7699a85a1a543c40bb72f6f06b8ae7e32fde2 WatchSource:0}: Error finding container 26d7cf6369c5e35d8b3193be45d7699a85a1a543c40bb72f6f06b8ae7e32fde2: Status 404 returned error can't find the container with id 26d7cf6369c5e35d8b3193be45d7699a85a1a543c40bb72f6f06b8ae7e32fde2 Jul 17 21:35:59.089763 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:59.089735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9343c441-c85c-4538-b70c-f843620e9dd0-webhook-cert\") pod \"maas-controller-5bf8dd86bf-v57mj\" (UID: \"9343c441-c85c-4538-b70c-f843620e9dd0\") " pod="opendatahub/maas-controller-5bf8dd86bf-v57mj" Jul 17 21:35:59.092256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:59.092234 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9343c441-c85c-4538-b70c-f843620e9dd0-webhook-cert\") pod \"maas-controller-5bf8dd86bf-v57mj\" (UID: \"9343c441-c85c-4538-b70c-f843620e9dd0\") " pod="opendatahub/maas-controller-5bf8dd86bf-v57mj" Jul 17 21:35:59.270219 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:59.270190 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bf8dd86bf-v57mj" Jul 17 21:35:59.434801 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:59.434577 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bf8dd86bf-v57mj"] Jul 17 21:35:59.437067 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:35:59.437040 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9343c441_c85c_4538_b70c_f843620e9dd0.slice/crio-259b8ad0a19e0696be64f887f1068fe5859525b4432329bb18ca95c3c3284ec6 WatchSource:0}: Error finding container 259b8ad0a19e0696be64f887f1068fe5859525b4432329bb18ca95c3c3284ec6: Status 404 returned error can't find the container with id 259b8ad0a19e0696be64f887f1068fe5859525b4432329bb18ca95c3c3284ec6 Jul 17 21:35:59.479146 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:59.479111 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bf8dd86bf-v57mj" event={"ID":"9343c441-c85c-4538-b70c-f843620e9dd0","Type":"ContainerStarted","Data":"259b8ad0a19e0696be64f887f1068fe5859525b4432329bb18ca95c3c3284ec6"} Jul 17 21:35:59.480348 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:35:59.480327 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86d8757495-9mqvf" event={"ID":"6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a","Type":"ContainerStarted","Data":"26d7cf6369c5e35d8b3193be45d7699a85a1a543c40bb72f6f06b8ae7e32fde2"} Jul 17 21:36:01.935354 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:01.935299 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:36:02.034274 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:02.034239 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:36:02.502234 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:02.502146 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bf8dd86bf-v57mj" event={"ID":"9343c441-c85c-4538-b70c-f843620e9dd0","Type":"ContainerStarted","Data":"5c5a23870e26df19cfffe02294620433216052cae895f058b5db220d4859f50c"} Jul 17 21:36:02.502452 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:02.502252 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bf8dd86bf-v57mj" Jul 17 21:36:02.503721 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:02.503694 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86d8757495-9mqvf" event={"ID":"6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a","Type":"ContainerStarted","Data":"203b875a085426cac9828e838eea17f76dfa48da780a9ace998d528414347ac0"} Jul 17 21:36:02.503837 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:02.503823 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86d8757495-9mqvf" Jul 17 21:36:02.520286 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:02.520233 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5bf8dd86bf-v57mj" podStartSLOduration=2.216716103 podStartE2EDuration="4.520216031s" totalImagesPullingTime="2.303499928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:36:02.517760207 +0000 UTC m=+503.939479460" watchObservedRunningTime="2026-07-17 21:36:02.520216031 +0000 UTC m=+503.941935265" Jul 17 21:36:02.533962 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:02.533917 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-86d8757495-9mqvf" podStartSLOduration=1.823060124 podStartE2EDuration="4.533901732s" totalImagesPullingTime="2.710841608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:36:02.533571838 +0000 UTC m=+503.955291090" watchObservedRunningTime="2026-07-17 21:36:02.533901732 +0000 UTC m=+503.955620967" Jul 17 21:36:13.510754 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:13.510725 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bf8dd86bf-v57mj" Jul 17 21:36:13.511149 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:13.511015 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86d8757495-9mqvf" Jul 17 21:36:13.569660 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:13.569626 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bf8dd86bf-v57mj"] Jul 17 21:36:13.569901 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:13.569840 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5bf8dd86bf-v57mj" podUID="9343c441-c85c-4538-b70c-f843620e9dd0" containerName="manager" containerID="cri-o://5c5a23870e26df19cfffe02294620433216052cae895f058b5db220d4859f50c" gracePeriod=30 Jul 17 21:36:13.820889 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:13.820858 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bf8dd86bf-v57mj" Jul 17 21:36:13.929641 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:13.929614 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9343c441-c85c-4538-b70c-f843620e9dd0-webhook-cert\") pod \"9343c441-c85c-4538-b70c-f843620e9dd0\" (UID: \"9343c441-c85c-4538-b70c-f843620e9dd0\") " Jul 17 21:36:13.929820 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:13.929687 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58g94\" (UniqueName: \"kubernetes.io/projected/9343c441-c85c-4538-b70c-f843620e9dd0-kube-api-access-58g94\") pod \"9343c441-c85c-4538-b70c-f843620e9dd0\" (UID: \"9343c441-c85c-4538-b70c-f843620e9dd0\") " Jul 17 21:36:13.932068 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:13.932030 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9343c441-c85c-4538-b70c-f843620e9dd0-kube-api-access-58g94" (OuterVolumeSpecName: "kube-api-access-58g94") pod "9343c441-c85c-4538-b70c-f843620e9dd0" (UID: "9343c441-c85c-4538-b70c-f843620e9dd0"). InnerVolumeSpecName "kube-api-access-58g94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:36:13.932068 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:13.932032 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9343c441-c85c-4538-b70c-f843620e9dd0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9343c441-c85c-4538-b70c-f843620e9dd0" (UID: "9343c441-c85c-4538-b70c-f843620e9dd0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:36:14.030469 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:14.030413 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9343c441-c85c-4538-b70c-f843620e9dd0-webhook-cert\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:36:14.030469 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:14.030460 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58g94\" (UniqueName: \"kubernetes.io/projected/9343c441-c85c-4538-b70c-f843620e9dd0-kube-api-access-58g94\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:36:14.567483 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:14.567450 2580 generic.go:356] "Generic (PLEG): container finished" podID="9343c441-c85c-4538-b70c-f843620e9dd0" containerID="5c5a23870e26df19cfffe02294620433216052cae895f058b5db220d4859f50c" exitCode=0 Jul 17 21:36:14.567872 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:14.567498 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bf8dd86bf-v57mj" Jul 17 21:36:14.567872 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:14.567541 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bf8dd86bf-v57mj" event={"ID":"9343c441-c85c-4538-b70c-f843620e9dd0","Type":"ContainerDied","Data":"5c5a23870e26df19cfffe02294620433216052cae895f058b5db220d4859f50c"} Jul 17 21:36:14.567872 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:14.567588 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bf8dd86bf-v57mj" event={"ID":"9343c441-c85c-4538-b70c-f843620e9dd0","Type":"ContainerDied","Data":"259b8ad0a19e0696be64f887f1068fe5859525b4432329bb18ca95c3c3284ec6"} Jul 17 21:36:14.567872 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:14.567610 2580 scope.go:117] "RemoveContainer" containerID="5c5a23870e26df19cfffe02294620433216052cae895f058b5db220d4859f50c" Jul 17 21:36:14.577076 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:14.576924 2580 scope.go:117] "RemoveContainer" containerID="5c5a23870e26df19cfffe02294620433216052cae895f058b5db220d4859f50c" Jul 17 21:36:14.577216 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:36:14.577199 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c5a23870e26df19cfffe02294620433216052cae895f058b5db220d4859f50c\": container with ID starting with 5c5a23870e26df19cfffe02294620433216052cae895f058b5db220d4859f50c not found: ID does not exist" containerID="5c5a23870e26df19cfffe02294620433216052cae895f058b5db220d4859f50c" Jul 17 21:36:14.577263 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:14.577225 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c5a23870e26df19cfffe02294620433216052cae895f058b5db220d4859f50c"} err="failed to get container status \"5c5a23870e26df19cfffe02294620433216052cae895f058b5db220d4859f50c\": rpc error: code = NotFound desc = could not find container \"5c5a23870e26df19cfffe02294620433216052cae895f058b5db220d4859f50c\": container with ID starting with 5c5a23870e26df19cfffe02294620433216052cae895f058b5db220d4859f50c not found: ID does not exist" Jul 17 21:36:14.588773 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:14.588748 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bf8dd86bf-v57mj"] Jul 17 21:36:14.596550 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:14.594825 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5bf8dd86bf-v57mj"] Jul 17 21:36:15.024351 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:15.024306 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9343c441-c85c-4538-b70c-f843620e9dd0" path="/var/lib/kubelet/pods/9343c441-c85c-4538-b70c-f843620e9dd0/volumes" Jul 17 21:36:22.629992 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:22.629956 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:36:31.319151 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:31.319066 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:36:31.419530 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:31.419492 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:36:31.822654 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:31.822616 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:36:32.103177 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.103095 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7d47bfd4-vbd66"] Jul 17 21:36:32.103471 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.103457 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9343c441-c85c-4538-b70c-f843620e9dd0" containerName="manager" Jul 17 21:36:32.103518 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.103472 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9343c441-c85c-4538-b70c-f843620e9dd0" containerName="manager" Jul 17 21:36:32.103564 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.103555 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9343c441-c85c-4538-b70c-f843620e9dd0" containerName="manager" Jul 17 21:36:32.106258 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.106242 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d47bfd4-vbd66" Jul 17 21:36:32.108765 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.108744 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 17 21:36:32.109779 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.109757 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rmn2m\"" Jul 17 21:36:32.109885 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.109825 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 21:36:32.109885 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.109842 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 21:36:32.117607 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.117582 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d47bfd4-vbd66"] Jul 17 21:36:32.176041 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.176004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/636cf20d-4276-45e6-87ec-c8d24c30724b-maas-api-tls\") pod \"maas-api-7d47bfd4-vbd66\" (UID: \"636cf20d-4276-45e6-87ec-c8d24c30724b\") " pod="odh-ai-gateway-infra/maas-api-7d47bfd4-vbd66" Jul 17 21:36:32.176232 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.176047 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/636cf20d-4276-45e6-87ec-c8d24c30724b-openshift-service-ca\") pod \"maas-api-7d47bfd4-vbd66\" (UID: \"636cf20d-4276-45e6-87ec-c8d24c30724b\") " pod="odh-ai-gateway-infra/maas-api-7d47bfd4-vbd66" Jul 17 21:36:32.176232 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.176169 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znqkm\" (UniqueName: \"kubernetes.io/projected/636cf20d-4276-45e6-87ec-c8d24c30724b-kube-api-access-znqkm\") pod \"maas-api-7d47bfd4-vbd66\" (UID: \"636cf20d-4276-45e6-87ec-c8d24c30724b\") " pod="odh-ai-gateway-infra/maas-api-7d47bfd4-vbd66" Jul 17 21:36:32.276716 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.276664 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znqkm\" (UniqueName: \"kubernetes.io/projected/636cf20d-4276-45e6-87ec-c8d24c30724b-kube-api-access-znqkm\") pod \"maas-api-7d47bfd4-vbd66\" (UID: \"636cf20d-4276-45e6-87ec-c8d24c30724b\") " pod="odh-ai-gateway-infra/maas-api-7d47bfd4-vbd66" Jul 17 21:36:32.276716 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.276719 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/636cf20d-4276-45e6-87ec-c8d24c30724b-maas-api-tls\") pod \"maas-api-7d47bfd4-vbd66\" (UID: \"636cf20d-4276-45e6-87ec-c8d24c30724b\") " pod="odh-ai-gateway-infra/maas-api-7d47bfd4-vbd66" Jul 17 21:36:32.276923 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.276736 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/636cf20d-4276-45e6-87ec-c8d24c30724b-openshift-service-ca\") pod \"maas-api-7d47bfd4-vbd66\" (UID: \"636cf20d-4276-45e6-87ec-c8d24c30724b\") " pod="odh-ai-gateway-infra/maas-api-7d47bfd4-vbd66" Jul 17 21:36:32.277453 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.277433 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/636cf20d-4276-45e6-87ec-c8d24c30724b-openshift-service-ca\") pod \"maas-api-7d47bfd4-vbd66\" (UID: \"636cf20d-4276-45e6-87ec-c8d24c30724b\") " pod="odh-ai-gateway-infra/maas-api-7d47bfd4-vbd66" Jul 17 21:36:32.279413 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.279353 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/636cf20d-4276-45e6-87ec-c8d24c30724b-maas-api-tls\") pod \"maas-api-7d47bfd4-vbd66\" (UID: \"636cf20d-4276-45e6-87ec-c8d24c30724b\") " pod="odh-ai-gateway-infra/maas-api-7d47bfd4-vbd66" Jul 17 21:36:32.284350 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.284323 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znqkm\" (UniqueName: \"kubernetes.io/projected/636cf20d-4276-45e6-87ec-c8d24c30724b-kube-api-access-znqkm\") pod \"maas-api-7d47bfd4-vbd66\" (UID: \"636cf20d-4276-45e6-87ec-c8d24c30724b\") " pod="odh-ai-gateway-infra/maas-api-7d47bfd4-vbd66" Jul 17 21:36:32.318567 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.318536 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:36:32.418200 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.418106 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d47bfd4-vbd66" Jul 17 21:36:32.522109 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.522075 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:36:32.569957 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.569933 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d47bfd4-vbd66"] Jul 17 21:36:32.572189 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:36:32.572159 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636cf20d_4276_45e6_87ec_c8d24c30724b.slice/crio-7c4d1c9ff7ecbe44f045c9806584201bd03b986b774938d564e9ee6952ded303 WatchSource:0}: Error finding container 7c4d1c9ff7ecbe44f045c9806584201bd03b986b774938d564e9ee6952ded303: Status 404 returned error can't find the container with id 7c4d1c9ff7ecbe44f045c9806584201bd03b986b774938d564e9ee6952ded303 Jul 17 21:36:32.615202 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.615171 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:36:32.661899 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:32.661861 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d47bfd4-vbd66" event={"ID":"636cf20d-4276-45e6-87ec-c8d24c30724b","Type":"ContainerStarted","Data":"7c4d1c9ff7ecbe44f045c9806584201bd03b986b774938d564e9ee6952ded303"} Jul 17 21:36:34.674304 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:34.674266 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d47bfd4-vbd66" event={"ID":"636cf20d-4276-45e6-87ec-c8d24c30724b","Type":"ContainerStarted","Data":"e12dc2ef57d665c4424d857dc9cdfe9de8c18f7cf464a81ce4421a4265008255"} Jul 17 21:36:34.674907 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:34.674380 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7d47bfd4-vbd66" Jul 17 21:36:34.693523 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:34.693469 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7d47bfd4-vbd66" podStartSLOduration=1.247683378 podStartE2EDuration="2.693454427s" totalImagesPullingTime="1.445771049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:36:34.691915805 +0000 UTC m=+536.113635046" watchObservedRunningTime="2026-07-17 21:36:34.693454427 +0000 UTC m=+536.115173706" Jul 17 21:36:40.684952 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:40.684918 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7d47bfd4-vbd66" Jul 17 21:36:41.679627 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:41.679591 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:36:41.766272 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:41.766238 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:36:47.451042 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:47.451006 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:36:49.845290 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:49.845250 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:36:49.912513 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:49.912484 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:36:52.404456 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.404412 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl"] Jul 17 21:36:52.411667 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.411643 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.414346 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.414313 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 17 21:36:52.414540 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.414517 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ghszk\"" Jul 17 21:36:52.414729 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.414710 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 17 21:36:52.415576 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.415558 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 17 21:36:52.418073 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.418047 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl"] Jul 17 21:36:52.429061 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.429033 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:36:52.463314 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.463271 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb2954c-6a9a-498f-857a-18259d4402b3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fmzfl\" (UID: \"7fb2954c-6a9a-498f-857a-18259d4402b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.463314 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.463312 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb2954c-6a9a-498f-857a-18259d4402b3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fmzfl\" (UID: \"7fb2954c-6a9a-498f-857a-18259d4402b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.463552 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.463374 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb2954c-6a9a-498f-857a-18259d4402b3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fmzfl\" (UID: \"7fb2954c-6a9a-498f-857a-18259d4402b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.463552 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.463446 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpg7r\" (UniqueName: \"kubernetes.io/projected/7fb2954c-6a9a-498f-857a-18259d4402b3-kube-api-access-lpg7r\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fmzfl\" (UID: \"7fb2954c-6a9a-498f-857a-18259d4402b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.463552 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.463481 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb2954c-6a9a-498f-857a-18259d4402b3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fmzfl\" (UID: \"7fb2954c-6a9a-498f-857a-18259d4402b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.463552 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.463514 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb2954c-6a9a-498f-857a-18259d4402b3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fmzfl\" (UID: \"7fb2954c-6a9a-498f-857a-18259d4402b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.564203 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.564165 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb2954c-6a9a-498f-857a-18259d4402b3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fmzfl\" (UID: \"7fb2954c-6a9a-498f-857a-18259d4402b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.564203 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.564206 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb2954c-6a9a-498f-857a-18259d4402b3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fmzfl\" (UID: \"7fb2954c-6a9a-498f-857a-18259d4402b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.564517 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.564235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb2954c-6a9a-498f-857a-18259d4402b3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fmzfl\" (UID: \"7fb2954c-6a9a-498f-857a-18259d4402b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.564517 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.564269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpg7r\" (UniqueName: \"kubernetes.io/projected/7fb2954c-6a9a-498f-857a-18259d4402b3-kube-api-access-lpg7r\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fmzfl\" (UID: \"7fb2954c-6a9a-498f-857a-18259d4402b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.564517 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.564310 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb2954c-6a9a-498f-857a-18259d4402b3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fmzfl\" (UID: \"7fb2954c-6a9a-498f-857a-18259d4402b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.564517 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.564345 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb2954c-6a9a-498f-857a-18259d4402b3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fmzfl\" (UID: \"7fb2954c-6a9a-498f-857a-18259d4402b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.564730 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.564708 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb2954c-6a9a-498f-857a-18259d4402b3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fmzfl\" (UID: \"7fb2954c-6a9a-498f-857a-18259d4402b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.564785 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.564744 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb2954c-6a9a-498f-857a-18259d4402b3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fmzfl\" (UID: \"7fb2954c-6a9a-498f-857a-18259d4402b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.564785 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.564770 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb2954c-6a9a-498f-857a-18259d4402b3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fmzfl\" (UID: \"7fb2954c-6a9a-498f-857a-18259d4402b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.566900 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.566878 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb2954c-6a9a-498f-857a-18259d4402b3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fmzfl\" (UID: \"7fb2954c-6a9a-498f-857a-18259d4402b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.567117 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.567097 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb2954c-6a9a-498f-857a-18259d4402b3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fmzfl\" (UID: \"7fb2954c-6a9a-498f-857a-18259d4402b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.572804 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.572782 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpg7r\" (UniqueName: \"kubernetes.io/projected/7fb2954c-6a9a-498f-857a-18259d4402b3-kube-api-access-lpg7r\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fmzfl\" (UID: \"7fb2954c-6a9a-498f-857a-18259d4402b3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.724157 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.724057 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:36:52.860594 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:52.860566 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl"] Jul 17 21:36:52.861903 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:36:52.861872 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb2954c_6a9a_498f_857a_18259d4402b3.slice/crio-181d9ce405c3ba5e709c2faf51668ecef9ead9ced4c0608c3f030f95d9e6f245 WatchSource:0}: Error finding container 181d9ce405c3ba5e709c2faf51668ecef9ead9ced4c0608c3f030f95d9e6f245: Status 404 returned error can't find the container with id 181d9ce405c3ba5e709c2faf51668ecef9ead9ced4c0608c3f030f95d9e6f245 Jul 17 21:36:53.775406 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:53.775344 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" event={"ID":"7fb2954c-6a9a-498f-857a-18259d4402b3","Type":"ContainerStarted","Data":"181d9ce405c3ba5e709c2faf51668ecef9ead9ced4c0608c3f030f95d9e6f245"} Jul 17 21:36:54.783285 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:54.783252 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" event={"ID":"7fb2954c-6a9a-498f-857a-18259d4402b3","Type":"ContainerStarted","Data":"66b1c120a561862d213cb0ee575959727bd1ec7f1d401bd44d52fa7ff646edb8"} Jul 17 21:36:54.802717 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:36:54.802666 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" podStartSLOduration=1.716017369 podStartE2EDuration="2.802649766s" totalImagesPullingTime="1.086632397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:36:54.800782864 +0000 UTC m=+556.222502108" watchObservedRunningTime="2026-07-17 21:36:54.802649766 +0000 UTC m=+556.224368998" Jul 17 21:37:02.725188 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:02.725143 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:37:02.725685 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:02.725218 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:37:02.738353 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:02.738326 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:37:02.834540 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:02.834506 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fmzfl" Jul 17 21:37:10.248862 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:10.248794 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:37:38.944556 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:38.944532 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 21:37:38.945720 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:38.945699 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 21:37:44.418696 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.418658 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789"] Jul 17 21:37:44.422017 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.421987 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.424504 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.424482 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 17 21:37:44.435315 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.435270 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789"] Jul 17 21:37:44.448672 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.448637 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:37:44.523581 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.523538 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e142f0b-c204-4279-ae86-259f295dfd78-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789\" (UID: \"0e142f0b-c204-4279-ae86-259f295dfd78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.523581 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.523588 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e142f0b-c204-4279-ae86-259f295dfd78-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789\" (UID: \"0e142f0b-c204-4279-ae86-259f295dfd78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.523815 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.523610 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e142f0b-c204-4279-ae86-259f295dfd78-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789\" (UID: \"0e142f0b-c204-4279-ae86-259f295dfd78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.523815 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.523704 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e142f0b-c204-4279-ae86-259f295dfd78-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789\" (UID: \"0e142f0b-c204-4279-ae86-259f295dfd78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.523815 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.523736 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw5w6\" (UniqueName: \"kubernetes.io/projected/0e142f0b-c204-4279-ae86-259f295dfd78-kube-api-access-zw5w6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789\" (UID: \"0e142f0b-c204-4279-ae86-259f295dfd78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.523815 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.523783 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e142f0b-c204-4279-ae86-259f295dfd78-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789\" (UID: \"0e142f0b-c204-4279-ae86-259f295dfd78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.625172 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.625128 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw5w6\" (UniqueName: \"kubernetes.io/projected/0e142f0b-c204-4279-ae86-259f295dfd78-kube-api-access-zw5w6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789\" (UID: \"0e142f0b-c204-4279-ae86-259f295dfd78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.625455 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.625198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e142f0b-c204-4279-ae86-259f295dfd78-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789\" (UID: \"0e142f0b-c204-4279-ae86-259f295dfd78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.625455 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.625268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e142f0b-c204-4279-ae86-259f295dfd78-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789\" (UID: \"0e142f0b-c204-4279-ae86-259f295dfd78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.625455 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.625308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e142f0b-c204-4279-ae86-259f295dfd78-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789\" (UID: \"0e142f0b-c204-4279-ae86-259f295dfd78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.625455 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.625336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e142f0b-c204-4279-ae86-259f295dfd78-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789\" (UID: \"0e142f0b-c204-4279-ae86-259f295dfd78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.625455 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.625369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e142f0b-c204-4279-ae86-259f295dfd78-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789\" (UID: \"0e142f0b-c204-4279-ae86-259f295dfd78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.625775 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.625752 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e142f0b-c204-4279-ae86-259f295dfd78-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789\" (UID: \"0e142f0b-c204-4279-ae86-259f295dfd78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.625839 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.625810 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e142f0b-c204-4279-ae86-259f295dfd78-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789\" (UID: \"0e142f0b-c204-4279-ae86-259f295dfd78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.625921 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.625902 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e142f0b-c204-4279-ae86-259f295dfd78-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789\" (UID: \"0e142f0b-c204-4279-ae86-259f295dfd78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.627797 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.627773 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e142f0b-c204-4279-ae86-259f295dfd78-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789\" (UID: \"0e142f0b-c204-4279-ae86-259f295dfd78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.628028 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.628013 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e142f0b-c204-4279-ae86-259f295dfd78-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789\" (UID: \"0e142f0b-c204-4279-ae86-259f295dfd78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.634419 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.634366 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw5w6\" (UniqueName: \"kubernetes.io/projected/0e142f0b-c204-4279-ae86-259f295dfd78-kube-api-access-zw5w6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789\" (UID: \"0e142f0b-c204-4279-ae86-259f295dfd78\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.733769 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.733682 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:44.875723 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:44.875692 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789"] Jul 17 21:37:44.877366 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:37:44.877338 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e142f0b_c204_4279_ae86_259f295dfd78.slice/crio-edfabbd14496badd8b2c012a9ec9e1f51fc1ed2a05f96f4b6f8255ba2707e9d0 WatchSource:0}: Error finding container edfabbd14496badd8b2c012a9ec9e1f51fc1ed2a05f96f4b6f8255ba2707e9d0: Status 404 returned error can't find the container with id edfabbd14496badd8b2c012a9ec9e1f51fc1ed2a05f96f4b6f8255ba2707e9d0 Jul 17 21:37:45.038935 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:45.038899 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" event={"ID":"0e142f0b-c204-4279-ae86-259f295dfd78","Type":"ContainerStarted","Data":"edfabbd14496badd8b2c012a9ec9e1f51fc1ed2a05f96f4b6f8255ba2707e9d0"} Jul 17 21:37:46.045696 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:46.045662 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" event={"ID":"0e142f0b-c204-4279-ae86-259f295dfd78","Type":"ContainerStarted","Data":"f32e47f90f0563bced5b9542e664b5c1a2cc15337a0b655fbb07011228960c5d"} Jul 17 21:37:46.081410 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:46.081327 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" podStartSLOduration=1.917825717 podStartE2EDuration="2.081303793s" totalImagesPullingTime="163.478076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:37:46.078051542 +0000 UTC m=+607.499770776" watchObservedRunningTime="2026-07-17 21:37:46.081303793 +0000 UTC m=+607.503023027" Jul 17 21:37:46.222968 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:46.222930 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:37:49.733987 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.733950 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66"] Jul 17 21:37:49.738131 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.738108 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:49.741997 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.741971 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 17 21:37:49.749838 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.749812 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66"] Jul 17 21:37:49.772841 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.772807 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:37:49.873483 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.873448 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59c1a244-a0d1-4e65-9c4e-919ff3da1fd4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66\" (UID: \"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:49.873657 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.873498 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c1a244-a0d1-4e65-9c4e-919ff3da1fd4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66\" (UID: \"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:49.873657 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.873562 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c1a244-a0d1-4e65-9c4e-919ff3da1fd4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66\" (UID: \"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:49.873657 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.873613 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59c1a244-a0d1-4e65-9c4e-919ff3da1fd4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66\" (UID: \"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:49.873781 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.873665 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmh5v\" (UniqueName: \"kubernetes.io/projected/59c1a244-a0d1-4e65-9c4e-919ff3da1fd4-kube-api-access-cmh5v\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66\" (UID: \"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:49.873781 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.873704 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59c1a244-a0d1-4e65-9c4e-919ff3da1fd4-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66\" (UID: \"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:49.974528 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.974494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmh5v\" (UniqueName: \"kubernetes.io/projected/59c1a244-a0d1-4e65-9c4e-919ff3da1fd4-kube-api-access-cmh5v\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66\" (UID: \"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:49.974726 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.974543 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59c1a244-a0d1-4e65-9c4e-919ff3da1fd4-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66\" (UID: \"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:49.974726 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.974584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59c1a244-a0d1-4e65-9c4e-919ff3da1fd4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66\" (UID: \"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:49.974726 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.974622 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c1a244-a0d1-4e65-9c4e-919ff3da1fd4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66\" (UID: \"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:49.974726 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.974653 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c1a244-a0d1-4e65-9c4e-919ff3da1fd4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66\" (UID: \"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:49.974726 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.974705 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59c1a244-a0d1-4e65-9c4e-919ff3da1fd4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66\" (UID: \"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:49.975082 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.975057 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59c1a244-a0d1-4e65-9c4e-919ff3da1fd4-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66\" (UID: \"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:49.975186 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.975107 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c1a244-a0d1-4e65-9c4e-919ff3da1fd4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66\" (UID: \"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:49.975186 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.975126 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59c1a244-a0d1-4e65-9c4e-919ff3da1fd4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66\" (UID: \"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:49.977483 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.977455 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59c1a244-a0d1-4e65-9c4e-919ff3da1fd4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66\" (UID: \"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:49.977603 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.977537 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c1a244-a0d1-4e65-9c4e-919ff3da1fd4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66\" (UID: \"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:49.986978 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:49.986919 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmh5v\" (UniqueName: \"kubernetes.io/projected/59c1a244-a0d1-4e65-9c4e-919ff3da1fd4-kube-api-access-cmh5v\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66\" (UID: \"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:50.049205 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:50.049172 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:37:50.195852 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:50.195822 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66"] Jul 17 21:37:50.198048 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:37:50.198017 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59c1a244_a0d1_4e65_9c4e_919ff3da1fd4.slice/crio-692eaca676f73348c7b7899ed2027f2e213bf21740d6bf349be14126e732f53e WatchSource:0}: Error finding container 692eaca676f73348c7b7899ed2027f2e213bf21740d6bf349be14126e732f53e: Status 404 returned error can't find the container with id 692eaca676f73348c7b7899ed2027f2e213bf21740d6bf349be14126e732f53e Jul 17 21:37:51.079668 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:51.079620 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" event={"ID":"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4","Type":"ContainerStarted","Data":"c6f5d46c7f58a490d32baddd46bdbbf4fb355eff8938560ac02f61da6bd5b09f"} Jul 17 21:37:51.079668 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:51.079674 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" event={"ID":"59c1a244-a0d1-4e65-9c4e-919ff3da1fd4","Type":"ContainerStarted","Data":"692eaca676f73348c7b7899ed2027f2e213bf21740d6bf349be14126e732f53e"} Jul 17 21:37:51.106159 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:51.106090 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" podStartSLOduration=1.9202743629999999 podStartE2EDuration="2.106065817s" totalImagesPullingTime="185.791454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:37:51.10276408 +0000 UTC m=+612.524483315" watchObservedRunningTime="2026-07-17 21:37:51.106065817 +0000 UTC m=+612.527785051" Jul 17 21:37:54.734849 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:54.734810 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:54.735238 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:54.734885 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:54.747546 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:54.747515 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:37:55.113417 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:37:55.113373 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789" Jul 17 21:38:00.049539 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:38:00.049450 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:38:00.049539 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:38:00.049508 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:38:00.062101 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:38:00.062075 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:38:00.137859 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:38:00.137823 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66" Jul 17 21:39:00.867800 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:00.867764 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-689b579cb6-vzbzb"] Jul 17 21:39:00.871312 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:00.871294 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-689b579cb6-vzbzb" Jul 17 21:39:00.895549 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:00.895523 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-689b579cb6-vzbzb"] Jul 17 21:39:00.924935 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:00.924898 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkhgr\" (UniqueName: \"kubernetes.io/projected/5da48d3c-c7f0-4a60-96fd-361218f717bb-kube-api-access-tkhgr\") pod \"authorino-689b579cb6-vzbzb\" (UID: \"5da48d3c-c7f0-4a60-96fd-361218f717bb\") " pod="kuadrant-system/authorino-689b579cb6-vzbzb" Jul 17 21:39:00.925105 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:00.924945 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5da48d3c-c7f0-4a60-96fd-361218f717bb-tls-cert\") pod \"authorino-689b579cb6-vzbzb\" (UID: \"5da48d3c-c7f0-4a60-96fd-361218f717bb\") " pod="kuadrant-system/authorino-689b579cb6-vzbzb" Jul 17 21:39:01.025407 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:01.025362 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkhgr\" (UniqueName: \"kubernetes.io/projected/5da48d3c-c7f0-4a60-96fd-361218f717bb-kube-api-access-tkhgr\") pod \"authorino-689b579cb6-vzbzb\" (UID: \"5da48d3c-c7f0-4a60-96fd-361218f717bb\") " pod="kuadrant-system/authorino-689b579cb6-vzbzb" Jul 17 21:39:01.025602 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:01.025420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5da48d3c-c7f0-4a60-96fd-361218f717bb-tls-cert\") pod \"authorino-689b579cb6-vzbzb\" (UID: \"5da48d3c-c7f0-4a60-96fd-361218f717bb\") " pod="kuadrant-system/authorino-689b579cb6-vzbzb" Jul 17 21:39:01.028180 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:01.028153 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5da48d3c-c7f0-4a60-96fd-361218f717bb-tls-cert\") pod \"authorino-689b579cb6-vzbzb\" (UID: \"5da48d3c-c7f0-4a60-96fd-361218f717bb\") " pod="kuadrant-system/authorino-689b579cb6-vzbzb" Jul 17 21:39:01.043602 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:01.043571 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkhgr\" (UniqueName: \"kubernetes.io/projected/5da48d3c-c7f0-4a60-96fd-361218f717bb-kube-api-access-tkhgr\") pod \"authorino-689b579cb6-vzbzb\" (UID: \"5da48d3c-c7f0-4a60-96fd-361218f717bb\") " pod="kuadrant-system/authorino-689b579cb6-vzbzb" Jul 17 21:39:01.181216 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:01.181113 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-689b579cb6-vzbzb" Jul 17 21:39:01.321981 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:01.321953 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:39:01.333675 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:01.333647 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-689b579cb6-vzbzb"] Jul 17 21:39:01.333832 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:39:01.333748 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5da48d3c_c7f0_4a60_96fd_361218f717bb.slice/crio-f22c84b38994f8fbc09e3a4a98b5b98d101f14e70006e83b5fb4c55d92f8a1a9 WatchSource:0}: Error finding container f22c84b38994f8fbc09e3a4a98b5b98d101f14e70006e83b5fb4c55d92f8a1a9: Status 404 returned error can't find the container with id f22c84b38994f8fbc09e3a4a98b5b98d101f14e70006e83b5fb4c55d92f8a1a9 Jul 17 21:39:01.335055 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:01.335040 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 21:39:01.451871 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:01.451787 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-689b579cb6-vzbzb" event={"ID":"5da48d3c-c7f0-4a60-96fd-361218f717bb","Type":"ContainerStarted","Data":"f22c84b38994f8fbc09e3a4a98b5b98d101f14e70006e83b5fb4c55d92f8a1a9"} Jul 17 21:39:01.458156 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:01.458127 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:39:02.459679 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:02.459637 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-689b579cb6-vzbzb" event={"ID":"5da48d3c-c7f0-4a60-96fd-361218f717bb","Type":"ContainerStarted","Data":"7d5486bbb18e1b523dd6bbbaac4644c800366e10a52abdd43fe77e05bd9b24ea"} Jul 17 21:39:02.503497 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:02.503436 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-689b579cb6-vzbzb" podStartSLOduration=2.061985748 podStartE2EDuration="2.503414139s" totalImagesPullingTime="441.428391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:39:02.503269733 +0000 UTC m=+683.924988979" watchObservedRunningTime="2026-07-17 21:39:02.503414139 +0000 UTC m=+683.925133374" Jul 17 21:39:02.572879 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:02.572845 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c885449c9-j8jgr"] Jul 17 21:39:02.573110 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:02.573066 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c885449c9-j8jgr" podUID="a1462236-b596-4a88-b3c1-9fa2ad76f382" containerName="authorino" containerID="cri-o://e6dffc6b18ee6dbc572effe390e231108978cde2f07b3829616517a7ded33021" gracePeriod=30 Jul 17 21:39:02.593224 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:02.593189 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:39:02.702635 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:02.702598 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:39:02.842868 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:02.842838 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c885449c9-j8jgr" Jul 17 21:39:02.944653 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:02.944612 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brb4z\" (UniqueName: \"kubernetes.io/projected/a1462236-b596-4a88-b3c1-9fa2ad76f382-kube-api-access-brb4z\") pod \"a1462236-b596-4a88-b3c1-9fa2ad76f382\" (UID: \"a1462236-b596-4a88-b3c1-9fa2ad76f382\") " Jul 17 21:39:02.944837 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:02.944718 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1462236-b596-4a88-b3c1-9fa2ad76f382-tls-cert\") pod \"a1462236-b596-4a88-b3c1-9fa2ad76f382\" (UID: \"a1462236-b596-4a88-b3c1-9fa2ad76f382\") " Jul 17 21:39:02.946812 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:02.946785 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1462236-b596-4a88-b3c1-9fa2ad76f382-kube-api-access-brb4z" (OuterVolumeSpecName: "kube-api-access-brb4z") pod "a1462236-b596-4a88-b3c1-9fa2ad76f382" (UID: "a1462236-b596-4a88-b3c1-9fa2ad76f382"). InnerVolumeSpecName "kube-api-access-brb4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:39:02.955592 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:02.955564 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1462236-b596-4a88-b3c1-9fa2ad76f382-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a1462236-b596-4a88-b3c1-9fa2ad76f382" (UID: "a1462236-b596-4a88-b3c1-9fa2ad76f382"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:39:03.046236 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:03.046204 2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1462236-b596-4a88-b3c1-9fa2ad76f382-tls-cert\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:39:03.046236 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:03.046234 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brb4z\" (UniqueName: \"kubernetes.io/projected/a1462236-b596-4a88-b3c1-9fa2ad76f382-kube-api-access-brb4z\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:39:03.466014 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:03.465913 2580 generic.go:356] "Generic (PLEG): container finished" podID="a1462236-b596-4a88-b3c1-9fa2ad76f382" containerID="e6dffc6b18ee6dbc572effe390e231108978cde2f07b3829616517a7ded33021" exitCode=0 Jul 17 21:39:03.466014 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:03.465950 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c885449c9-j8jgr" Jul 17 21:39:03.466014 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:03.466000 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c885449c9-j8jgr" event={"ID":"a1462236-b596-4a88-b3c1-9fa2ad76f382","Type":"ContainerDied","Data":"e6dffc6b18ee6dbc572effe390e231108978cde2f07b3829616517a7ded33021"} Jul 17 21:39:03.466562 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:03.466042 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c885449c9-j8jgr" event={"ID":"a1462236-b596-4a88-b3c1-9fa2ad76f382","Type":"ContainerDied","Data":"2730135de4519368b265524595b7749f5abbaf792e320b105c29c429c8700dcf"} Jul 17 21:39:03.466562 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:03.466059 2580 scope.go:117] "RemoveContainer" containerID="e6dffc6b18ee6dbc572effe390e231108978cde2f07b3829616517a7ded33021" Jul 17 21:39:03.478028 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:03.478005 2580 scope.go:117] "RemoveContainer" containerID="e6dffc6b18ee6dbc572effe390e231108978cde2f07b3829616517a7ded33021" Jul 17 21:39:03.478350 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:39:03.478327 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6dffc6b18ee6dbc572effe390e231108978cde2f07b3829616517a7ded33021\": container with ID starting with e6dffc6b18ee6dbc572effe390e231108978cde2f07b3829616517a7ded33021 not found: ID does not exist" containerID="e6dffc6b18ee6dbc572effe390e231108978cde2f07b3829616517a7ded33021" Jul 17 21:39:03.478463 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:03.478370 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6dffc6b18ee6dbc572effe390e231108978cde2f07b3829616517a7ded33021"} err="failed to get container status \"e6dffc6b18ee6dbc572effe390e231108978cde2f07b3829616517a7ded33021\": rpc error: code = NotFound desc = could not find container \"e6dffc6b18ee6dbc572effe390e231108978cde2f07b3829616517a7ded33021\": container with ID starting with e6dffc6b18ee6dbc572effe390e231108978cde2f07b3829616517a7ded33021 not found: ID does not exist" Jul 17 21:39:03.503091 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:03.503049 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c885449c9-j8jgr"] Jul 17 21:39:03.518127 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:03.518093 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c885449c9-j8jgr"] Jul 17 21:39:05.024369 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:05.024333 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1462236-b596-4a88-b3c1-9fa2ad76f382" path="/var/lib/kubelet/pods/a1462236-b596-4a88-b3c1-9fa2ad76f382/volumes" Jul 17 21:39:26.638876 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:26.638833 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:39:48.667694 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:48.667653 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86d8757495-9mqvf"] Jul 17 21:39:48.668186 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:48.667906 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-86d8757495-9mqvf" podUID="6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a" containerName="manager" containerID="cri-o://203b875a085426cac9828e838eea17f76dfa48da780a9ace998d528414347ac0" gracePeriod=30 Jul 17 21:39:48.918013 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:48.917952 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86d8757495-9mqvf" Jul 17 21:39:48.970904 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:48.970872 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsvs5\" (UniqueName: \"kubernetes.io/projected/6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a-kube-api-access-zsvs5\") pod \"6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a\" (UID: \"6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a\") " Jul 17 21:39:48.971100 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:48.971018 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a-webhook-cert\") pod \"6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a\" (UID: \"6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a\") " Jul 17 21:39:48.973135 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:48.973103 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a-kube-api-access-zsvs5" (OuterVolumeSpecName: "kube-api-access-zsvs5") pod "6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a" (UID: "6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a"). InnerVolumeSpecName "kube-api-access-zsvs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:39:48.973254 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:48.973180 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a" (UID: "6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:39:49.072534 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:49.072504 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsvs5\" (UniqueName: \"kubernetes.io/projected/6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a-kube-api-access-zsvs5\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:39:49.072534 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:49.072533 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a-webhook-cert\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:39:49.710337 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:49.710300 2580 generic.go:356] "Generic (PLEG): container finished" podID="6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a" containerID="203b875a085426cac9828e838eea17f76dfa48da780a9ace998d528414347ac0" exitCode=0 Jul 17 21:39:49.710823 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:49.710348 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86d8757495-9mqvf" Jul 17 21:39:49.710823 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:49.710386 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86d8757495-9mqvf" event={"ID":"6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a","Type":"ContainerDied","Data":"203b875a085426cac9828e838eea17f76dfa48da780a9ace998d528414347ac0"} Jul 17 21:39:49.710823 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:49.710453 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86d8757495-9mqvf" event={"ID":"6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a","Type":"ContainerDied","Data":"26d7cf6369c5e35d8b3193be45d7699a85a1a543c40bb72f6f06b8ae7e32fde2"} Jul 17 21:39:49.710823 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:49.710475 2580 scope.go:117] "RemoveContainer" containerID="203b875a085426cac9828e838eea17f76dfa48da780a9ace998d528414347ac0" Jul 17 21:39:49.719587 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:49.719568 2580 scope.go:117] "RemoveContainer" containerID="203b875a085426cac9828e838eea17f76dfa48da780a9ace998d528414347ac0" Jul 17 21:39:49.719868 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:39:49.719847 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"203b875a085426cac9828e838eea17f76dfa48da780a9ace998d528414347ac0\": container with ID starting with 203b875a085426cac9828e838eea17f76dfa48da780a9ace998d528414347ac0 not found: ID does not exist" containerID="203b875a085426cac9828e838eea17f76dfa48da780a9ace998d528414347ac0" Jul 17 21:39:49.719939 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:49.719876 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"203b875a085426cac9828e838eea17f76dfa48da780a9ace998d528414347ac0"} err="failed to get container status \"203b875a085426cac9828e838eea17f76dfa48da780a9ace998d528414347ac0\": rpc error: code = NotFound desc = could not find container \"203b875a085426cac9828e838eea17f76dfa48da780a9ace998d528414347ac0\": container with ID starting with 203b875a085426cac9828e838eea17f76dfa48da780a9ace998d528414347ac0 not found: ID does not exist" Jul 17 21:39:49.731650 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:49.731625 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86d8757495-9mqvf"] Jul 17 21:39:49.744079 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:49.744055 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86d8757495-9mqvf"] Jul 17 21:39:51.024072 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:39:51.024032 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a" path="/var/lib/kubelet/pods/6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a/volumes" Jul 17 21:40:50.560198 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:50.560160 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86d8757495-h27xs"] Jul 17 21:40:50.560657 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:50.560618 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a" containerName="manager" Jul 17 21:40:50.560657 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:50.560631 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a" containerName="manager" Jul 17 21:40:50.560657 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:50.560649 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1462236-b596-4a88-b3c1-9fa2ad76f382" containerName="authorino" Jul 17 21:40:50.560657 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:50.560655 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1462236-b596-4a88-b3c1-9fa2ad76f382" containerName="authorino" Jul 17 21:40:50.560794 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:50.560712 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1462236-b596-4a88-b3c1-9fa2ad76f382" containerName="authorino" Jul 17 21:40:50.560794 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:50.560722 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e46030f-efe6-4b4e-bd65-a4a3a11dcf0a" containerName="manager" Jul 17 21:40:50.565173 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:50.565155 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86d8757495-h27xs" Jul 17 21:40:50.568948 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:50.568926 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 17 21:40:50.569081 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:50.568970 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 17 21:40:50.569081 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:50.568970 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-n7k8g\"" Jul 17 21:40:50.573956 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:50.573933 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86d8757495-h27xs"] Jul 17 21:40:50.732690 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:50.732652 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81594114-0922-4bbd-b58d-f1979d56194d-webhook-cert\") pod \"maas-controller-86d8757495-h27xs\" (UID: \"81594114-0922-4bbd-b58d-f1979d56194d\") " pod="opendatahub/maas-controller-86d8757495-h27xs" Jul 17 21:40:50.732871 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:50.732738 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnmfs\" (UniqueName: \"kubernetes.io/projected/81594114-0922-4bbd-b58d-f1979d56194d-kube-api-access-hnmfs\") pod \"maas-controller-86d8757495-h27xs\" (UID: \"81594114-0922-4bbd-b58d-f1979d56194d\") " pod="opendatahub/maas-controller-86d8757495-h27xs" Jul 17 21:40:50.833784 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:50.833681 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnmfs\" (UniqueName: \"kubernetes.io/projected/81594114-0922-4bbd-b58d-f1979d56194d-kube-api-access-hnmfs\") pod \"maas-controller-86d8757495-h27xs\" (UID: \"81594114-0922-4bbd-b58d-f1979d56194d\") " pod="opendatahub/maas-controller-86d8757495-h27xs" Jul 17 21:40:50.833784 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:50.833777 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81594114-0922-4bbd-b58d-f1979d56194d-webhook-cert\") pod \"maas-controller-86d8757495-h27xs\" (UID: \"81594114-0922-4bbd-b58d-f1979d56194d\") " pod="opendatahub/maas-controller-86d8757495-h27xs" Jul 17 21:40:50.836378 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:50.836358 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81594114-0922-4bbd-b58d-f1979d56194d-webhook-cert\") pod \"maas-controller-86d8757495-h27xs\" (UID: \"81594114-0922-4bbd-b58d-f1979d56194d\") " pod="opendatahub/maas-controller-86d8757495-h27xs" Jul 17 21:40:50.843017 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:50.842992 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnmfs\" (UniqueName: \"kubernetes.io/projected/81594114-0922-4bbd-b58d-f1979d56194d-kube-api-access-hnmfs\") pod \"maas-controller-86d8757495-h27xs\" (UID: \"81594114-0922-4bbd-b58d-f1979d56194d\") " pod="opendatahub/maas-controller-86d8757495-h27xs" Jul 17 21:40:50.876256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:50.876229 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86d8757495-h27xs" Jul 17 21:40:51.019834 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:40:51.019805 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81594114_0922_4bbd_b58d_f1979d56194d.slice/crio-33bbd7ff4700667323bf6264b4476d6838f80ae3fbf932b347a1d707bfbb6b0c WatchSource:0}: Error finding container 33bbd7ff4700667323bf6264b4476d6838f80ae3fbf932b347a1d707bfbb6b0c: Status 404 returned error can't find the container with id 33bbd7ff4700667323bf6264b4476d6838f80ae3fbf932b347a1d707bfbb6b0c Jul 17 21:40:51.025271 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:51.025249 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86d8757495-h27xs"] Jul 17 21:40:51.040721 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:51.040692 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86d8757495-h27xs" event={"ID":"81594114-0922-4bbd-b58d-f1979d56194d","Type":"ContainerStarted","Data":"33bbd7ff4700667323bf6264b4476d6838f80ae3fbf932b347a1d707bfbb6b0c"} Jul 17 21:40:52.048149 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:52.048113 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86d8757495-h27xs" event={"ID":"81594114-0922-4bbd-b58d-f1979d56194d","Type":"ContainerStarted","Data":"43d0045a2fd093903d287b7337ec98390ce3b05158e3105b78bed67539387195"} Jul 17 21:40:52.048576 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:52.048272 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86d8757495-h27xs" Jul 17 21:40:52.067437 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:40:52.067375 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-86d8757495-h27xs" podStartSLOduration=1.608979278 podStartE2EDuration="2.067360349s" totalImagesPullingTime="458.381071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:40:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:40:52.064741213 +0000 UTC m=+793.486460448" watchObservedRunningTime="2026-07-17 21:40:52.067360349 +0000 UTC m=+793.489079581" Jul 17 21:41:03.056216 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:03.056148 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86d8757495-h27xs" Jul 17 21:41:07.637993 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:07.637957 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:41:07.711991 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:07.711957 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86d8757495-h27xs"] Jul 17 21:41:07.712187 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:07.712154 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-86d8757495-h27xs" podUID="81594114-0922-4bbd-b58d-f1979d56194d" containerName="manager" containerID="cri-o://43d0045a2fd093903d287b7337ec98390ce3b05158e3105b78bed67539387195" gracePeriod=30 Jul 17 21:41:08.058157 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.058125 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86d8757495-h27xs" Jul 17 21:41:08.101316 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.101286 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81594114-0922-4bbd-b58d-f1979d56194d-webhook-cert\") pod \"81594114-0922-4bbd-b58d-f1979d56194d\" (UID: \"81594114-0922-4bbd-b58d-f1979d56194d\") " Jul 17 21:41:08.101494 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.101345 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnmfs\" (UniqueName: \"kubernetes.io/projected/81594114-0922-4bbd-b58d-f1979d56194d-kube-api-access-hnmfs\") pod \"81594114-0922-4bbd-b58d-f1979d56194d\" (UID: \"81594114-0922-4bbd-b58d-f1979d56194d\") " Jul 17 21:41:08.103594 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.103566 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81594114-0922-4bbd-b58d-f1979d56194d-kube-api-access-hnmfs" (OuterVolumeSpecName: "kube-api-access-hnmfs") pod "81594114-0922-4bbd-b58d-f1979d56194d" (UID: "81594114-0922-4bbd-b58d-f1979d56194d"). InnerVolumeSpecName "kube-api-access-hnmfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:41:08.103726 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.103593 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81594114-0922-4bbd-b58d-f1979d56194d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "81594114-0922-4bbd-b58d-f1979d56194d" (UID: "81594114-0922-4bbd-b58d-f1979d56194d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:41:08.133502 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.133468 2580 generic.go:356] "Generic (PLEG): container finished" podID="81594114-0922-4bbd-b58d-f1979d56194d" containerID="43d0045a2fd093903d287b7337ec98390ce3b05158e3105b78bed67539387195" exitCode=0 Jul 17 21:41:08.133643 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.133512 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86d8757495-h27xs" Jul 17 21:41:08.133643 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.133558 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86d8757495-h27xs" event={"ID":"81594114-0922-4bbd-b58d-f1979d56194d","Type":"ContainerDied","Data":"43d0045a2fd093903d287b7337ec98390ce3b05158e3105b78bed67539387195"} Jul 17 21:41:08.133715 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.133645 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86d8757495-h27xs" event={"ID":"81594114-0922-4bbd-b58d-f1979d56194d","Type":"ContainerDied","Data":"33bbd7ff4700667323bf6264b4476d6838f80ae3fbf932b347a1d707bfbb6b0c"} Jul 17 21:41:08.133715 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.133670 2580 scope.go:117] "RemoveContainer" containerID="43d0045a2fd093903d287b7337ec98390ce3b05158e3105b78bed67539387195" Jul 17 21:41:08.143222 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.143203 2580 scope.go:117] "RemoveContainer" containerID="43d0045a2fd093903d287b7337ec98390ce3b05158e3105b78bed67539387195" Jul 17 21:41:08.143530 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:41:08.143511 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d0045a2fd093903d287b7337ec98390ce3b05158e3105b78bed67539387195\": container with ID starting with 43d0045a2fd093903d287b7337ec98390ce3b05158e3105b78bed67539387195 not found: ID does not exist" containerID="43d0045a2fd093903d287b7337ec98390ce3b05158e3105b78bed67539387195" Jul 17 21:41:08.143581 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.143541 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d0045a2fd093903d287b7337ec98390ce3b05158e3105b78bed67539387195"} err="failed to get container status \"43d0045a2fd093903d287b7337ec98390ce3b05158e3105b78bed67539387195\": rpc error: code = NotFound desc = could not find container \"43d0045a2fd093903d287b7337ec98390ce3b05158e3105b78bed67539387195\": container with ID starting with 43d0045a2fd093903d287b7337ec98390ce3b05158e3105b78bed67539387195 not found: ID does not exist" Jul 17 21:41:08.165302 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.165268 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86d8757495-h27xs"] Jul 17 21:41:08.184742 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.184673 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86d8757495-h27xs"] Jul 17 21:41:08.202332 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.202308 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnmfs\" (UniqueName: \"kubernetes.io/projected/81594114-0922-4bbd-b58d-f1979d56194d-kube-api-access-hnmfs\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:41:08.202433 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.202332 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81594114-0922-4bbd-b58d-f1979d56194d-webhook-cert\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:41:08.846993 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.846959 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:41:08.956437 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.956407 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86d8757495-sqn2m"] Jul 17 21:41:08.956777 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.956765 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81594114-0922-4bbd-b58d-f1979d56194d" containerName="manager" Jul 17 21:41:08.956821 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.956779 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="81594114-0922-4bbd-b58d-f1979d56194d" containerName="manager" Jul 17 21:41:08.956855 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.956841 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="81594114-0922-4bbd-b58d-f1979d56194d" containerName="manager" Jul 17 21:41:08.961151 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.961132 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86d8757495-sqn2m" Jul 17 21:41:08.964815 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.964785 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-n7k8g\"" Jul 17 21:41:08.964815 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.964800 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 17 21:41:08.964988 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.964859 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 17 21:41:08.969605 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:08.969582 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86d8757495-sqn2m"] Jul 17 21:41:09.010477 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:09.010438 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpsvv\" (UniqueName: \"kubernetes.io/projected/0f885ea0-1e53-4a18-975d-be0ea8fe68ed-kube-api-access-xpsvv\") pod \"maas-controller-86d8757495-sqn2m\" (UID: \"0f885ea0-1e53-4a18-975d-be0ea8fe68ed\") " pod="opendatahub/maas-controller-86d8757495-sqn2m" Jul 17 21:41:09.010666 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:09.010509 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f885ea0-1e53-4a18-975d-be0ea8fe68ed-webhook-cert\") pod \"maas-controller-86d8757495-sqn2m\" (UID: \"0f885ea0-1e53-4a18-975d-be0ea8fe68ed\") " pod="opendatahub/maas-controller-86d8757495-sqn2m" Jul 17 21:41:09.024767 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:09.024736 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81594114-0922-4bbd-b58d-f1979d56194d" path="/var/lib/kubelet/pods/81594114-0922-4bbd-b58d-f1979d56194d/volumes" Jul 17 21:41:09.111182 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:09.111091 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpsvv\" (UniqueName: \"kubernetes.io/projected/0f885ea0-1e53-4a18-975d-be0ea8fe68ed-kube-api-access-xpsvv\") pod \"maas-controller-86d8757495-sqn2m\" (UID: \"0f885ea0-1e53-4a18-975d-be0ea8fe68ed\") " pod="opendatahub/maas-controller-86d8757495-sqn2m" Jul 17 21:41:09.111182 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:09.111150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f885ea0-1e53-4a18-975d-be0ea8fe68ed-webhook-cert\") pod \"maas-controller-86d8757495-sqn2m\" (UID: \"0f885ea0-1e53-4a18-975d-be0ea8fe68ed\") " pod="opendatahub/maas-controller-86d8757495-sqn2m" Jul 17 21:41:09.113839 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:09.113817 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f885ea0-1e53-4a18-975d-be0ea8fe68ed-webhook-cert\") pod \"maas-controller-86d8757495-sqn2m\" (UID: \"0f885ea0-1e53-4a18-975d-be0ea8fe68ed\") " pod="opendatahub/maas-controller-86d8757495-sqn2m" Jul 17 21:41:09.119874 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:09.119851 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpsvv\" (UniqueName: \"kubernetes.io/projected/0f885ea0-1e53-4a18-975d-be0ea8fe68ed-kube-api-access-xpsvv\") pod \"maas-controller-86d8757495-sqn2m\" (UID: \"0f885ea0-1e53-4a18-975d-be0ea8fe68ed\") " pod="opendatahub/maas-controller-86d8757495-sqn2m" Jul 17 21:41:09.272002 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:09.271971 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86d8757495-sqn2m" Jul 17 21:41:09.412133 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:09.412107 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86d8757495-sqn2m"] Jul 17 21:41:09.414181 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:41:09.414150 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f885ea0_1e53_4a18_975d_be0ea8fe68ed.slice/crio-bb9f2765fd054de85d93a13f24bf3b75aee24358b1d6ebe03fc27c3c0adfedb1 WatchSource:0}: Error finding container bb9f2765fd054de85d93a13f24bf3b75aee24358b1d6ebe03fc27c3c0adfedb1: Status 404 returned error can't find the container with id bb9f2765fd054de85d93a13f24bf3b75aee24358b1d6ebe03fc27c3c0adfedb1 Jul 17 21:41:10.148619 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:10.148517 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86d8757495-sqn2m" event={"ID":"0f885ea0-1e53-4a18-975d-be0ea8fe68ed","Type":"ContainerStarted","Data":"4b256f96d3185950f5c8b08c19dccbfe79e7427a269861cc9e6d8a32768638ff"} Jul 17 21:41:10.148619 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:10.148579 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86d8757495-sqn2m" event={"ID":"0f885ea0-1e53-4a18-975d-be0ea8fe68ed","Type":"ContainerStarted","Data":"bb9f2765fd054de85d93a13f24bf3b75aee24358b1d6ebe03fc27c3c0adfedb1"} Jul 17 21:41:10.148619 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:10.148606 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86d8757495-sqn2m" Jul 17 21:41:10.168986 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:10.168934 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-86d8757495-sqn2m" podStartSLOduration=1.815153426 podStartE2EDuration="2.168919829s" totalImagesPullingTime="353.766403ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:41:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:41:10.167759965 +0000 UTC m=+811.589479199" watchObservedRunningTime="2026-07-17 21:41:10.168919829 +0000 UTC m=+811.590639123" Jul 17 21:41:21.157739 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:21.157692 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86d8757495-sqn2m" Jul 17 21:41:49.436117 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:49.436084 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:41:49.535235 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:49.535201 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:41:49.657019 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:49.656983 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:41:55.694761 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:55.694722 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:41:55.805224 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:55.805192 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:41:55.895199 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:55.895168 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:41:56.072860 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:41:56.072824 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:42:01.803595 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:42:01.803563 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:42:01.864283 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:42:01.864249 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:42:30.820477 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:42:30.820378 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:42:33.497094 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:42:33.497064 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:42:33.564646 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:42:33.564600 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:42:35.810993 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:42:35.810932 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:42:38.977293 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:42:38.977268 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 21:42:38.980859 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:42:38.980841 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 21:44:07.490085 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:44:07.490044 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:44:18.012150 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:44:18.012113 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:46:24.371064 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:46:24.371025 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:46:24.432123 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:46:24.432080 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:47:39.024683 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:47:39.024654 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 21:47:39.026619 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:47:39.026594 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 21:51:46.816957 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:51:46.816868 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:51:56.526215 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:51:56.526178 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:52:05.426903 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:52:05.426860 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:52:39.068292 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:52:39.068249 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 21:52:39.070951 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:52:39.070928 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 21:52:52.721260 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:52:52.721214 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:53:00.175858 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:00.175765 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:53:08.451513 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:08.451472 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:53:13.532803 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:13.532766 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:53:16.514504 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:16.514463 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:53:16.615892 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:16.615848 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:53:16.956962 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:16.956874 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:53:22.747745 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:22.747703 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:53:22.806265 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:22.806225 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:53:22.956712 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:22.956678 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:53:23.058223 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:23.058190 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:53:24.065364 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.065330 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:53:24.146835 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.146804 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q"] Jul 17 21:53:24.151025 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.151001 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:53:24.153345 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.153322 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f9bc6369-serving-cert\"" Jul 17 21:53:24.159029 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.158865 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q"] Jul 17 21:53:24.164270 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.164247 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:53:24.227786 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.227751 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:53:24.319793 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.319704 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a28a25a4-e7d2-42da-bb32-ba5495582af8-openshift-service-ca\") pod \"maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q\" (UID: \"a28a25a4-e7d2-42da-bb32-ba5495582af8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:53:24.319793 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.319747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7wn9\" (UniqueName: \"kubernetes.io/projected/a28a25a4-e7d2-42da-bb32-ba5495582af8-kube-api-access-b7wn9\") pod \"maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q\" (UID: \"a28a25a4-e7d2-42da-bb32-ba5495582af8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:53:24.319793 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.319790 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a28a25a4-e7d2-42da-bb32-ba5495582af8-maas-api-tls\") pod \"maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q\" (UID: \"a28a25a4-e7d2-42da-bb32-ba5495582af8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:53:24.420625 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.420584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a28a25a4-e7d2-42da-bb32-ba5495582af8-openshift-service-ca\") pod \"maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q\" (UID: \"a28a25a4-e7d2-42da-bb32-ba5495582af8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:53:24.420625 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.420630 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7wn9\" (UniqueName: \"kubernetes.io/projected/a28a25a4-e7d2-42da-bb32-ba5495582af8-kube-api-access-b7wn9\") pod \"maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q\" (UID: \"a28a25a4-e7d2-42da-bb32-ba5495582af8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:53:24.420845 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.420684 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a28a25a4-e7d2-42da-bb32-ba5495582af8-maas-api-tls\") pod \"maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q\" (UID: \"a28a25a4-e7d2-42da-bb32-ba5495582af8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:53:24.421362 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.421335 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a28a25a4-e7d2-42da-bb32-ba5495582af8-openshift-service-ca\") pod \"maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q\" (UID: \"a28a25a4-e7d2-42da-bb32-ba5495582af8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:53:24.423504 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.423480 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a28a25a4-e7d2-42da-bb32-ba5495582af8-maas-api-tls\") pod \"maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q\" (UID: \"a28a25a4-e7d2-42da-bb32-ba5495582af8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:53:24.429160 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.429137 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7wn9\" (UniqueName: \"kubernetes.io/projected/a28a25a4-e7d2-42da-bb32-ba5495582af8-kube-api-access-b7wn9\") pod \"maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q\" (UID: \"a28a25a4-e7d2-42da-bb32-ba5495582af8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:53:24.463196 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.463166 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:53:24.468303 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.468266 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:53:24.600146 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.600119 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q"] Jul 17 21:53:24.602423 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:53:24.602372 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda28a25a4_e7d2_42da_bb32_ba5495582af8.slice/crio-989c347e1cd581a333268b7cffe47d1dc051981161f1648bfb71124189e131e2 WatchSource:0}: Error finding container 989c347e1cd581a333268b7cffe47d1dc051981161f1648bfb71124189e131e2: Status 404 returned error can't find the container with id 989c347e1cd581a333268b7cffe47d1dc051981161f1648bfb71124189e131e2 Jul 17 21:53:24.603853 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.603831 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 21:53:24.904881 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:24.904765 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:53:25.035359 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:25.035328 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:53:25.075219 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:25.075186 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" event={"ID":"a28a25a4-e7d2-42da-bb32-ba5495582af8","Type":"ContainerStarted","Data":"989c347e1cd581a333268b7cffe47d1dc051981161f1648bfb71124189e131e2"} Jul 17 21:53:26.084160 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:26.084128 2580 generic.go:356] "Generic (PLEG): container finished" podID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerID="c7df22d7de661606adc13de6e5b5569511055efdd98721b1dd10c4ded0b08244" exitCode=1 Jul 17 21:53:26.084566 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:26.084197 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" event={"ID":"a28a25a4-e7d2-42da-bb32-ba5495582af8","Type":"ContainerDied","Data":"c7df22d7de661606adc13de6e5b5569511055efdd98721b1dd10c4ded0b08244"} Jul 17 21:53:26.084566 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:26.084463 2580 scope.go:117] "RemoveContainer" containerID="c7df22d7de661606adc13de6e5b5569511055efdd98721b1dd10c4ded0b08244" Jul 17 21:53:26.855162 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:26.855123 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:53:27.091761 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:27.091672 2580 generic.go:356] "Generic (PLEG): container finished" podID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerID="a0198f5cc520632213c686ae13d0259eb2d2d54574b74f7651a54967c60fca0a" exitCode=1 Jul 17 21:53:27.092153 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:27.091773 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" event={"ID":"a28a25a4-e7d2-42da-bb32-ba5495582af8","Type":"ContainerDied","Data":"a0198f5cc520632213c686ae13d0259eb2d2d54574b74f7651a54967c60fca0a"} Jul 17 21:53:27.092153 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:27.091839 2580 scope.go:117] "RemoveContainer" containerID="c7df22d7de661606adc13de6e5b5569511055efdd98721b1dd10c4ded0b08244" Jul 17 21:53:27.092153 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:27.091973 2580 scope.go:117] "RemoveContainer" containerID="a0198f5cc520632213c686ae13d0259eb2d2d54574b74f7651a54967c60fca0a" Jul 17 21:53:27.092316 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:53:27.092207 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:53:28.098791 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:28.098768 2580 scope.go:117] "RemoveContainer" containerID="a0198f5cc520632213c686ae13d0259eb2d2d54574b74f7651a54967c60fca0a" Jul 17 21:53:28.099195 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:53:28.098957 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:53:29.464147 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:29.464118 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:53:29.464708 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:29.464691 2580 scope.go:117] "RemoveContainer" containerID="a0198f5cc520632213c686ae13d0259eb2d2d54574b74f7651a54967c60fca0a" Jul 17 21:53:29.464954 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:53:29.464927 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:53:34.463503 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:34.463466 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:53:34.463893 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:34.463859 2580 scope.go:117] "RemoveContainer" containerID="a0198f5cc520632213c686ae13d0259eb2d2d54574b74f7651a54967c60fca0a" Jul 17 21:53:34.464062 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:53:34.464041 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:53:45.018799 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:45.018765 2580 scope.go:117] "RemoveContainer" containerID="a0198f5cc520632213c686ae13d0259eb2d2d54574b74f7651a54967c60fca0a" Jul 17 21:53:46.199717 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:46.199621 2580 generic.go:356] "Generic (PLEG): container finished" podID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerID="9cecef916c2302ebbb5bd9fd160bafb4ceb2e691053b5bee8086e63274b8a602" exitCode=1 Jul 17 21:53:46.199717 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:46.199675 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" event={"ID":"a28a25a4-e7d2-42da-bb32-ba5495582af8","Type":"ContainerDied","Data":"9cecef916c2302ebbb5bd9fd160bafb4ceb2e691053b5bee8086e63274b8a602"} Jul 17 21:53:46.200184 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:46.199723 2580 scope.go:117] "RemoveContainer" containerID="a0198f5cc520632213c686ae13d0259eb2d2d54574b74f7651a54967c60fca0a" Jul 17 21:53:46.200184 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:46.200019 2580 scope.go:117] "RemoveContainer" containerID="9cecef916c2302ebbb5bd9fd160bafb4ceb2e691053b5bee8086e63274b8a602" Jul 17 21:53:46.200293 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:53:46.200221 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:53:49.464110 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:49.464072 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:53:49.464545 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:49.464471 2580 scope.go:117] "RemoveContainer" containerID="9cecef916c2302ebbb5bd9fd160bafb4ceb2e691053b5bee8086e63274b8a602" Jul 17 21:53:49.464657 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:53:49.464636 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:53:54.464074 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:54.464040 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:53:54.464615 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:53:54.464596 2580 scope.go:117] "RemoveContainer" containerID="9cecef916c2302ebbb5bd9fd160bafb4ceb2e691053b5bee8086e63274b8a602" Jul 17 21:53:54.464849 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:53:54.464824 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:54:05.018891 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:54:05.018855 2580 scope.go:117] "RemoveContainer" containerID="9cecef916c2302ebbb5bd9fd160bafb4ceb2e691053b5bee8086e63274b8a602" Jul 17 21:54:05.019415 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:54:05.019035 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:54:06.093083 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:54:06.093044 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:54:20.018626 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:54:20.018595 2580 scope.go:117] "RemoveContainer" containerID="9cecef916c2302ebbb5bd9fd160bafb4ceb2e691053b5bee8086e63274b8a602" Jul 17 21:54:21.384768 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:54:21.384736 2580 generic.go:356] "Generic (PLEG): container finished" podID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerID="ff455ab87b50d39ea60025ad61ee8da50b70406549dcf6f1108d50dac2f997ce" exitCode=1 Jul 17 21:54:21.385205 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:54:21.384804 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" event={"ID":"a28a25a4-e7d2-42da-bb32-ba5495582af8","Type":"ContainerDied","Data":"ff455ab87b50d39ea60025ad61ee8da50b70406549dcf6f1108d50dac2f997ce"} Jul 17 21:54:21.385205 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:54:21.384859 2580 scope.go:117] "RemoveContainer" containerID="9cecef916c2302ebbb5bd9fd160bafb4ceb2e691053b5bee8086e63274b8a602" Jul 17 21:54:21.385326 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:54:21.385224 2580 scope.go:117] "RemoveContainer" containerID="ff455ab87b50d39ea60025ad61ee8da50b70406549dcf6f1108d50dac2f997ce" Jul 17 21:54:21.385439 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:54:21.385422 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:54:24.463898 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:54:24.463851 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:54:24.464352 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:54:24.463945 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:54:24.464352 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:54:24.464253 2580 scope.go:117] "RemoveContainer" containerID="ff455ab87b50d39ea60025ad61ee8da50b70406549dcf6f1108d50dac2f997ce" Jul 17 21:54:24.464471 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:54:24.464435 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:54:40.018146 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:54:40.018062 2580 scope.go:117] "RemoveContainer" containerID="ff455ab87b50d39ea60025ad61ee8da50b70406549dcf6f1108d50dac2f997ce" Jul 17 21:54:40.018562 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:54:40.018241 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:54:55.018919 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:54:55.018877 2580 scope.go:117] "RemoveContainer" containerID="ff455ab87b50d39ea60025ad61ee8da50b70406549dcf6f1108d50dac2f997ce" Jul 17 21:54:55.019673 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:54:55.019120 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:55:08.018067 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:55:08.018027 2580 scope.go:117] "RemoveContainer" containerID="ff455ab87b50d39ea60025ad61ee8da50b70406549dcf6f1108d50dac2f997ce" Jul 17 21:55:08.651431 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:55:08.651372 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" event={"ID":"a28a25a4-e7d2-42da-bb32-ba5495582af8","Type":"ContainerStarted","Data":"011573e84a368e413d8f82da83aad4f6093e4d92f977504d6b6271c6f607b109"} Jul 17 21:55:08.651624 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:55:08.651607 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:55:08.671623 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:55:08.671564 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podStartSLOduration=102.475101678 podStartE2EDuration="1m44.671540808s" totalImagesPullingTime="2.19643913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:53:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:55:08.669757009 +0000 UTC m=+1650.091476268" watchObservedRunningTime="2026-07-17 21:55:08.671540808 +0000 UTC m=+1650.093260041" Jul 17 21:55:09.657945 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:55:09.657905 2580 generic.go:356] "Generic (PLEG): container finished" podID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerID="011573e84a368e413d8f82da83aad4f6093e4d92f977504d6b6271c6f607b109" exitCode=1 Jul 17 21:55:09.658469 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:55:09.658000 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" event={"ID":"a28a25a4-e7d2-42da-bb32-ba5495582af8","Type":"ContainerDied","Data":"011573e84a368e413d8f82da83aad4f6093e4d92f977504d6b6271c6f607b109"} Jul 17 21:55:09.658469 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:55:09.658086 2580 scope.go:117] "RemoveContainer" containerID="ff455ab87b50d39ea60025ad61ee8da50b70406549dcf6f1108d50dac2f997ce" Jul 17 21:55:09.658469 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:55:09.658295 2580 scope.go:117] "RemoveContainer" containerID="011573e84a368e413d8f82da83aad4f6093e4d92f977504d6b6271c6f607b109" Jul 17 21:55:09.658636 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:55:09.658539 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:55:10.665047 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:55:10.665017 2580 scope.go:117] "RemoveContainer" containerID="011573e84a368e413d8f82da83aad4f6093e4d92f977504d6b6271c6f607b109" Jul 17 21:55:10.665439 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:55:10.665248 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:55:14.464101 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:55:14.464067 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:55:14.464728 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:55:14.464566 2580 scope.go:117] "RemoveContainer" containerID="011573e84a368e413d8f82da83aad4f6093e4d92f977504d6b6271c6f607b109" Jul 17 21:55:14.464843 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:55:14.464816 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:55:26.018058 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:55:26.018015 2580 scope.go:117] "RemoveContainer" containerID="011573e84a368e413d8f82da83aad4f6093e4d92f977504d6b6271c6f607b109" Jul 17 21:55:26.018496 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:55:26.018220 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:55:39.027647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:55:39.027619 2580 scope.go:117] "RemoveContainer" containerID="011573e84a368e413d8f82da83aad4f6093e4d92f977504d6b6271c6f607b109" Jul 17 21:55:39.028077 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:55:39.027822 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:55:51.018510 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:55:51.018474 2580 scope.go:117] "RemoveContainer" containerID="011573e84a368e413d8f82da83aad4f6093e4d92f977504d6b6271c6f607b109" Jul 17 21:55:51.019115 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:55:51.018671 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:56:05.018366 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:05.018333 2580 scope.go:117] "RemoveContainer" containerID="011573e84a368e413d8f82da83aad4f6093e4d92f977504d6b6271c6f607b109" Jul 17 21:56:05.018816 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:56:05.018554 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:56:16.018577 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:16.018547 2580 scope.go:117] "RemoveContainer" containerID="011573e84a368e413d8f82da83aad4f6093e4d92f977504d6b6271c6f607b109" Jul 17 21:56:16.018977 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:56:16.018738 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:56:29.022543 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:29.022507 2580 scope.go:117] "RemoveContainer" containerID="011573e84a368e413d8f82da83aad4f6093e4d92f977504d6b6271c6f607b109" Jul 17 21:56:29.296750 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:29.296672 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:56:29.361628 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:29.361594 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:56:30.087106 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:30.087074 2580 generic.go:356] "Generic (PLEG): container finished" podID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerID="2af486a669023dd4c8dee2fb65820f76d7eb089b445686faae5ea018e54b1d73" exitCode=1 Jul 17 21:56:30.087539 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:30.087154 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" event={"ID":"a28a25a4-e7d2-42da-bb32-ba5495582af8","Type":"ContainerDied","Data":"2af486a669023dd4c8dee2fb65820f76d7eb089b445686faae5ea018e54b1d73"} Jul 17 21:56:30.087539 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:30.087220 2580 scope.go:117] "RemoveContainer" containerID="011573e84a368e413d8f82da83aad4f6093e4d92f977504d6b6271c6f607b109" Jul 17 21:56:30.087662 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:30.087651 2580 scope.go:117] "RemoveContainer" containerID="2af486a669023dd4c8dee2fb65820f76d7eb089b445686faae5ea018e54b1d73" Jul 17 21:56:30.087878 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:56:30.087860 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:56:30.886652 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:30.886622 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:56:30.988344 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:30.988308 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:56:33.652451 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:33.652416 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:56:33.652971 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:33.652926 2580 scope.go:117] "RemoveContainer" containerID="2af486a669023dd4c8dee2fb65820f76d7eb089b445686faae5ea018e54b1d73" Jul 17 21:56:33.653166 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:56:33.653139 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:56:34.463885 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:34.463849 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:56:34.464265 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:34.464252 2580 scope.go:117] "RemoveContainer" containerID="2af486a669023dd4c8dee2fb65820f76d7eb089b445686faae5ea018e54b1d73" Jul 17 21:56:34.464495 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:56:34.464473 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q_odh-ai-gateway-infra(a28a25a4-e7d2-42da-bb32-ba5495582af8)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" Jul 17 21:56:34.686091 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:34.686056 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q"] Jul 17 21:56:34.716494 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:34.716413 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:56:35.138872 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:35.138839 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:56:35.249997 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:35.249975 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:56:35.301501 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:35.301472 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7wn9\" (UniqueName: \"kubernetes.io/projected/a28a25a4-e7d2-42da-bb32-ba5495582af8-kube-api-access-b7wn9\") pod \"a28a25a4-e7d2-42da-bb32-ba5495582af8\" (UID: \"a28a25a4-e7d2-42da-bb32-ba5495582af8\") " Jul 17 21:56:35.301698 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:35.301553 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a28a25a4-e7d2-42da-bb32-ba5495582af8-openshift-service-ca\") pod \"a28a25a4-e7d2-42da-bb32-ba5495582af8\" (UID: \"a28a25a4-e7d2-42da-bb32-ba5495582af8\") " Jul 17 21:56:35.301698 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:35.301617 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a28a25a4-e7d2-42da-bb32-ba5495582af8-maas-api-tls\") pod \"a28a25a4-e7d2-42da-bb32-ba5495582af8\" (UID: \"a28a25a4-e7d2-42da-bb32-ba5495582af8\") " Jul 17 21:56:35.301921 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:35.301893 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a28a25a4-e7d2-42da-bb32-ba5495582af8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a28a25a4-e7d2-42da-bb32-ba5495582af8" (UID: "a28a25a4-e7d2-42da-bb32-ba5495582af8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:56:35.304015 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:35.303990 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a28a25a4-e7d2-42da-bb32-ba5495582af8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a28a25a4-e7d2-42da-bb32-ba5495582af8" (UID: "a28a25a4-e7d2-42da-bb32-ba5495582af8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:56:35.304204 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:35.304014 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a28a25a4-e7d2-42da-bb32-ba5495582af8-kube-api-access-b7wn9" (OuterVolumeSpecName: "kube-api-access-b7wn9") pod "a28a25a4-e7d2-42da-bb32-ba5495582af8" (UID: "a28a25a4-e7d2-42da-bb32-ba5495582af8"). InnerVolumeSpecName "kube-api-access-b7wn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:56:35.402925 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:35.402845 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a28a25a4-e7d2-42da-bb32-ba5495582af8-openshift-service-ca\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:56:35.402925 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:35.402877 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a28a25a4-e7d2-42da-bb32-ba5495582af8-maas-api-tls\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:56:35.402925 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:35.402886 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7wn9\" (UniqueName: \"kubernetes.io/projected/a28a25a4-e7d2-42da-bb32-ba5495582af8-kube-api-access-b7wn9\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:56:36.123106 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:36.123080 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" Jul 17 21:56:36.123106 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:36.123094 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q" event={"ID":"a28a25a4-e7d2-42da-bb32-ba5495582af8","Type":"ContainerDied","Data":"989c347e1cd581a333268b7cffe47d1dc051981161f1648bfb71124189e131e2"} Jul 17 21:56:36.123676 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:36.123152 2580 scope.go:117] "RemoveContainer" containerID="2af486a669023dd4c8dee2fb65820f76d7eb089b445686faae5ea018e54b1d73" Jul 17 21:56:36.148848 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:36.148808 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q"] Jul 17 21:56:36.151283 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:36.151261 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f9bc6369-66b87f6c47-t7z5q"] Jul 17 21:56:37.023588 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:37.023557 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" path="/var/lib/kubelet/pods/a28a25a4-e7d2-42da-bb32-ba5495582af8/volumes" Jul 17 21:56:40.789634 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:40.789603 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:56:40.889293 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:40.889260 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:56:41.071699 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:41.071611 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:56:41.168848 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:41.168814 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:56:41.282053 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:41.282014 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:56:42.547511 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.547477 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:56:42.724489 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.724447 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm"] Jul 17 21:56:42.724846 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.724827 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerName="maas-api" Jul 17 21:56:42.724929 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.724849 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerName="maas-api" Jul 17 21:56:42.724929 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.724863 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerName="maas-api" Jul 17 21:56:42.724929 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.724871 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerName="maas-api" Jul 17 21:56:42.724929 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.724883 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerName="maas-api" Jul 17 21:56:42.724929 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.724892 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerName="maas-api" Jul 17 21:56:42.724929 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.724906 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerName="maas-api" Jul 17 21:56:42.724929 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.724914 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerName="maas-api" Jul 17 21:56:42.725169 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.725015 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerName="maas-api" Jul 17 21:56:42.725169 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.725027 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerName="maas-api" Jul 17 21:56:42.725169 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.725039 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerName="maas-api" Jul 17 21:56:42.725169 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.725049 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerName="maas-api" Jul 17 21:56:42.729549 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.729520 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" Jul 17 21:56:42.732156 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.732131 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-281d4ac2-serving-cert\"" Jul 17 21:56:42.741175 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.741156 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm"] Jul 17 21:56:42.762095 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.762065 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:56:42.881193 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.881096 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c84a2810-3320-4044-ba33-3ae9df132f96-maas-api-tls\") pod \"maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm\" (UID: \"c84a2810-3320-4044-ba33-3ae9df132f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" Jul 17 21:56:42.881353 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.881229 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-855jl\" (UniqueName: \"kubernetes.io/projected/c84a2810-3320-4044-ba33-3ae9df132f96-kube-api-access-855jl\") pod \"maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm\" (UID: \"c84a2810-3320-4044-ba33-3ae9df132f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" Jul 17 21:56:42.881353 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.881282 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c84a2810-3320-4044-ba33-3ae9df132f96-openshift-service-ca\") pod \"maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm\" (UID: \"c84a2810-3320-4044-ba33-3ae9df132f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" Jul 17 21:56:42.981915 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.981878 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:56:42.982084 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.981929 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c84a2810-3320-4044-ba33-3ae9df132f96-maas-api-tls\") pod \"maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm\" (UID: \"c84a2810-3320-4044-ba33-3ae9df132f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" Jul 17 21:56:42.982084 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.982059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-855jl\" (UniqueName: \"kubernetes.io/projected/c84a2810-3320-4044-ba33-3ae9df132f96-kube-api-access-855jl\") pod \"maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm\" (UID: \"c84a2810-3320-4044-ba33-3ae9df132f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" Jul 17 21:56:42.982165 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.982119 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c84a2810-3320-4044-ba33-3ae9df132f96-openshift-service-ca\") pod \"maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm\" (UID: \"c84a2810-3320-4044-ba33-3ae9df132f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" Jul 17 21:56:42.982773 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.982757 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c84a2810-3320-4044-ba33-3ae9df132f96-openshift-service-ca\") pod \"maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm\" (UID: \"c84a2810-3320-4044-ba33-3ae9df132f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" Jul 17 21:56:42.984646 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.984628 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c84a2810-3320-4044-ba33-3ae9df132f96-maas-api-tls\") pod \"maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm\" (UID: \"c84a2810-3320-4044-ba33-3ae9df132f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" Jul 17 21:56:42.990876 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:42.990852 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-855jl\" (UniqueName: \"kubernetes.io/projected/c84a2810-3320-4044-ba33-3ae9df132f96-kube-api-access-855jl\") pod \"maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm\" (UID: \"c84a2810-3320-4044-ba33-3ae9df132f96\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" Jul 17 21:56:43.042131 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:43.042096 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" Jul 17 21:56:43.193596 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:43.193572 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm"] Jul 17 21:56:43.195823 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:56:43.195797 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84a2810_3320_4044_ba33_3ae9df132f96.slice/crio-d153c0356997ecd117b58b51c7e656fe5c6551b48a46da6b63c2fa865b421789 WatchSource:0}: Error finding container d153c0356997ecd117b58b51c7e656fe5c6551b48a46da6b63c2fa865b421789: Status 404 returned error can't find the container with id d153c0356997ecd117b58b51c7e656fe5c6551b48a46da6b63c2fa865b421789 Jul 17 21:56:43.213032 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:43.213002 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:56:43.312001 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:43.311962 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:56:44.170114 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:44.170007 2580 generic.go:356] "Generic (PLEG): container finished" podID="c84a2810-3320-4044-ba33-3ae9df132f96" containerID="04c104530da70922dcd62d2e2ae321f74e62417a63a6d27c1d5300216d247e73" exitCode=1 Jul 17 21:56:44.170114 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:44.170085 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" event={"ID":"c84a2810-3320-4044-ba33-3ae9df132f96","Type":"ContainerDied","Data":"04c104530da70922dcd62d2e2ae321f74e62417a63a6d27c1d5300216d247e73"} Jul 17 21:56:44.170600 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:44.170138 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" event={"ID":"c84a2810-3320-4044-ba33-3ae9df132f96","Type":"ContainerStarted","Data":"d153c0356997ecd117b58b51c7e656fe5c6551b48a46da6b63c2fa865b421789"} Jul 17 21:56:44.170600 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:44.170358 2580 scope.go:117] "RemoveContainer" containerID="04c104530da70922dcd62d2e2ae321f74e62417a63a6d27c1d5300216d247e73" Jul 17 21:56:45.177355 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:45.177276 2580 generic.go:356] "Generic (PLEG): container finished" podID="c84a2810-3320-4044-ba33-3ae9df132f96" containerID="ad5518f0642550c334927f70fa5a8489e133e134eeb1af96f512759f25edb69a" exitCode=1 Jul 17 21:56:45.177355 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:45.177338 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" event={"ID":"c84a2810-3320-4044-ba33-3ae9df132f96","Type":"ContainerDied","Data":"ad5518f0642550c334927f70fa5a8489e133e134eeb1af96f512759f25edb69a"} Jul 17 21:56:45.177822 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:45.177385 2580 scope.go:117] "RemoveContainer" containerID="04c104530da70922dcd62d2e2ae321f74e62417a63a6d27c1d5300216d247e73" Jul 17 21:56:45.177822 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:45.177601 2580 scope.go:117] "RemoveContainer" containerID="ad5518f0642550c334927f70fa5a8489e133e134eeb1af96f512759f25edb69a" Jul 17 21:56:45.177904 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:56:45.177832 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm_odh-ai-gateway-infra(c84a2810-3320-4044-ba33-3ae9df132f96)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" Jul 17 21:56:46.183801 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:46.183777 2580 scope.go:117] "RemoveContainer" containerID="ad5518f0642550c334927f70fa5a8489e133e134eeb1af96f512759f25edb69a" Jul 17 21:56:46.184245 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:56:46.184016 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm_odh-ai-gateway-infra(c84a2810-3320-4044-ba33-3ae9df132f96)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" Jul 17 21:56:48.043105 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:48.043067 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" Jul 17 21:56:48.043534 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:48.043516 2580 scope.go:117] "RemoveContainer" containerID="ad5518f0642550c334927f70fa5a8489e133e134eeb1af96f512759f25edb69a" Jul 17 21:56:48.043713 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:56:48.043692 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm_odh-ai-gateway-infra(c84a2810-3320-4044-ba33-3ae9df132f96)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" Jul 17 21:56:53.043053 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:53.043018 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" Jul 17 21:56:53.043464 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:56:53.043447 2580 scope.go:117] "RemoveContainer" containerID="ad5518f0642550c334927f70fa5a8489e133e134eeb1af96f512759f25edb69a" Jul 17 21:56:53.043655 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:56:53.043633 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm_odh-ai-gateway-infra(c84a2810-3320-4044-ba33-3ae9df132f96)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" Jul 17 21:57:05.018856 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:05.018818 2580 scope.go:117] "RemoveContainer" containerID="ad5518f0642550c334927f70fa5a8489e133e134eeb1af96f512759f25edb69a" Jul 17 21:57:06.289549 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:06.289515 2580 generic.go:356] "Generic (PLEG): container finished" podID="c84a2810-3320-4044-ba33-3ae9df132f96" containerID="81c3d6f8d343c34987790bf1cffe0eb8c2a8904f82a0a95209146e6b947196f4" exitCode=1 Jul 17 21:57:06.289997 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:06.289572 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" event={"ID":"c84a2810-3320-4044-ba33-3ae9df132f96","Type":"ContainerDied","Data":"81c3d6f8d343c34987790bf1cffe0eb8c2a8904f82a0a95209146e6b947196f4"} Jul 17 21:57:06.289997 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:06.289626 2580 scope.go:117] "RemoveContainer" containerID="ad5518f0642550c334927f70fa5a8489e133e134eeb1af96f512759f25edb69a" Jul 17 21:57:06.289997 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:06.289955 2580 scope.go:117] "RemoveContainer" containerID="81c3d6f8d343c34987790bf1cffe0eb8c2a8904f82a0a95209146e6b947196f4" Jul 17 21:57:06.290207 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:57:06.290188 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm_odh-ai-gateway-infra(c84a2810-3320-4044-ba33-3ae9df132f96)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" Jul 17 21:57:08.043298 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:08.043259 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" Jul 17 21:57:08.043819 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:08.043744 2580 scope.go:117] "RemoveContainer" containerID="81c3d6f8d343c34987790bf1cffe0eb8c2a8904f82a0a95209146e6b947196f4" Jul 17 21:57:08.043995 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:57:08.043970 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm_odh-ai-gateway-infra(c84a2810-3320-4044-ba33-3ae9df132f96)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" Jul 17 21:57:13.043366 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:13.043335 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" Jul 17 21:57:13.043808 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:13.043740 2580 scope.go:117] "RemoveContainer" containerID="81c3d6f8d343c34987790bf1cffe0eb8c2a8904f82a0a95209146e6b947196f4" Jul 17 21:57:13.043944 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:57:13.043922 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm_odh-ai-gateway-infra(c84a2810-3320-4044-ba33-3ae9df132f96)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" Jul 17 21:57:24.533425 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:24.533366 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:57:25.819058 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:25.819017 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:57:27.024116 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:27.024089 2580 scope.go:117] "RemoveContainer" containerID="81c3d6f8d343c34987790bf1cffe0eb8c2a8904f82a0a95209146e6b947196f4" Jul 17 21:57:28.408070 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:28.408039 2580 generic.go:356] "Generic (PLEG): container finished" podID="c84a2810-3320-4044-ba33-3ae9df132f96" containerID="5b8c2036047518cf1d4510101cbb5a9db55881bc9aa310099fd5aa67d91b2ded" exitCode=1 Jul 17 21:57:28.408526 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:28.408121 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" event={"ID":"c84a2810-3320-4044-ba33-3ae9df132f96","Type":"ContainerDied","Data":"5b8c2036047518cf1d4510101cbb5a9db55881bc9aa310099fd5aa67d91b2ded"} Jul 17 21:57:28.408526 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:28.408188 2580 scope.go:117] "RemoveContainer" containerID="81c3d6f8d343c34987790bf1cffe0eb8c2a8904f82a0a95209146e6b947196f4" Jul 17 21:57:28.408651 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:28.408621 2580 scope.go:117] "RemoveContainer" containerID="5b8c2036047518cf1d4510101cbb5a9db55881bc9aa310099fd5aa67d91b2ded" Jul 17 21:57:28.408846 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:57:28.408827 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm_odh-ai-gateway-infra(c84a2810-3320-4044-ba33-3ae9df132f96)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" Jul 17 21:57:33.043166 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:33.043075 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" Jul 17 21:57:33.043166 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:33.043144 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" Jul 17 21:57:33.043627 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:33.043518 2580 scope.go:117] "RemoveContainer" containerID="5b8c2036047518cf1d4510101cbb5a9db55881bc9aa310099fd5aa67d91b2ded" Jul 17 21:57:33.043725 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:57:33.043705 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm_odh-ai-gateway-infra(c84a2810-3320-4044-ba33-3ae9df132f96)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" Jul 17 21:57:39.103288 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:39.103258 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 21:57:39.107126 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:39.107105 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 21:57:44.018148 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:44.018115 2580 scope.go:117] "RemoveContainer" containerID="5b8c2036047518cf1d4510101cbb5a9db55881bc9aa310099fd5aa67d91b2ded" Jul 17 21:57:44.018569 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:57:44.018345 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm_odh-ai-gateway-infra(c84a2810-3320-4044-ba33-3ae9df132f96)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" Jul 17 21:57:47.172602 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:47.172565 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:57:47.277569 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:47.277538 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:57:51.006054 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:51.006005 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:57:52.543524 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:52.543483 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm"] Jul 17 21:57:52.574931 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:52.574891 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:57:52.677644 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:52.677620 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" Jul 17 21:57:52.721221 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:52.721185 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c84a2810-3320-4044-ba33-3ae9df132f96-openshift-service-ca\") pod \"c84a2810-3320-4044-ba33-3ae9df132f96\" (UID: \"c84a2810-3320-4044-ba33-3ae9df132f96\") " Jul 17 21:57:52.721221 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:52.721225 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-855jl\" (UniqueName: \"kubernetes.io/projected/c84a2810-3320-4044-ba33-3ae9df132f96-kube-api-access-855jl\") pod \"c84a2810-3320-4044-ba33-3ae9df132f96\" (UID: \"c84a2810-3320-4044-ba33-3ae9df132f96\") " Jul 17 21:57:52.721450 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:52.721248 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c84a2810-3320-4044-ba33-3ae9df132f96-maas-api-tls\") pod \"c84a2810-3320-4044-ba33-3ae9df132f96\" (UID: \"c84a2810-3320-4044-ba33-3ae9df132f96\") " Jul 17 21:57:52.721580 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:52.721555 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84a2810-3320-4044-ba33-3ae9df132f96-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c84a2810-3320-4044-ba33-3ae9df132f96" (UID: "c84a2810-3320-4044-ba33-3ae9df132f96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:57:52.723513 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:52.723482 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84a2810-3320-4044-ba33-3ae9df132f96-kube-api-access-855jl" (OuterVolumeSpecName: "kube-api-access-855jl") pod "c84a2810-3320-4044-ba33-3ae9df132f96" (UID: "c84a2810-3320-4044-ba33-3ae9df132f96"). InnerVolumeSpecName "kube-api-access-855jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:57:52.723513 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:52.723485 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84a2810-3320-4044-ba33-3ae9df132f96-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c84a2810-3320-4044-ba33-3ae9df132f96" (UID: "c84a2810-3320-4044-ba33-3ae9df132f96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:57:52.822740 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:52.822652 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c84a2810-3320-4044-ba33-3ae9df132f96-openshift-service-ca\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:57:52.822740 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:52.822684 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-855jl\" (UniqueName: \"kubernetes.io/projected/c84a2810-3320-4044-ba33-3ae9df132f96-kube-api-access-855jl\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:57:52.822740 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:52.822695 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c84a2810-3320-4044-ba33-3ae9df132f96-maas-api-tls\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 21:57:53.064775 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:53.064740 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:57:53.546114 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:53.546076 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" Jul 17 21:57:53.546114 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:53.546102 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm" event={"ID":"c84a2810-3320-4044-ba33-3ae9df132f96","Type":"ContainerDied","Data":"d153c0356997ecd117b58b51c7e656fe5c6551b48a46da6b63c2fa865b421789"} Jul 17 21:57:53.546666 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:53.546176 2580 scope.go:117] "RemoveContainer" containerID="5b8c2036047518cf1d4510101cbb5a9db55881bc9aa310099fd5aa67d91b2ded" Jul 17 21:57:53.564317 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:53.564285 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm"] Jul 17 21:57:53.566399 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:53.566364 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-281d4ac2-5c6ddf6895-lnktm"] Jul 17 21:57:55.024183 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:55.024146 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" path="/var/lib/kubelet/pods/c84a2810-3320-4044-ba33-3ae9df132f96/volumes" Jul 17 21:57:58.533026 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:58.532985 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:57:58.634959 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:58.634919 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:57:58.799090 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:58.799003 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:57:58.887100 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:58.887069 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:57:59.905857 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:57:59.905826 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:58:00.004426 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.004375 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:58:00.107828 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.107797 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx"] Jul 17 21:58:00.108166 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.108153 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" containerName="maas-api" Jul 17 21:58:00.108209 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.108168 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" containerName="maas-api" Jul 17 21:58:00.108209 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.108175 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" containerName="maas-api" Jul 17 21:58:00.108209 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.108180 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" containerName="maas-api" Jul 17 21:58:00.108209 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.108195 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" containerName="maas-api" Jul 17 21:58:00.108209 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.108200 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" containerName="maas-api" Jul 17 21:58:00.108209 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.108208 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerName="maas-api" Jul 17 21:58:00.108448 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.108213 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerName="maas-api" Jul 17 21:58:00.108448 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.108221 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerName="maas-api" Jul 17 21:58:00.108448 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.108226 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerName="maas-api" Jul 17 21:58:00.108448 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.108276 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerName="maas-api" Jul 17 21:58:00.108448 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.108282 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" containerName="maas-api" Jul 17 21:58:00.108448 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.108291 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" containerName="maas-api" Jul 17 21:58:00.108448 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.108297 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" containerName="maas-api" Jul 17 21:58:00.112986 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.112964 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 21:58:00.115607 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.115583 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e3cf3d26-serving-cert\"" Jul 17 21:58:00.122695 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.122666 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx"] Jul 17 21:58:00.171257 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.171172 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:58:00.184321 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.184292 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7gkh\" (UniqueName: \"kubernetes.io/projected/c0e0743c-3c3f-4283-936c-da9c4d2be4f3-kube-api-access-q7gkh\") pod \"maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx\" (UID: \"c0e0743c-3c3f-4283-936c-da9c4d2be4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 21:58:00.184501 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.184342 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0e0743c-3c3f-4283-936c-da9c4d2be4f3-maas-api-tls\") pod \"maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx\" (UID: \"c0e0743c-3c3f-4283-936c-da9c4d2be4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 21:58:00.184501 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.184431 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0e0743c-3c3f-4283-936c-da9c4d2be4f3-openshift-service-ca\") pod \"maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx\" (UID: \"c0e0743c-3c3f-4283-936c-da9c4d2be4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 21:58:00.285588 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.285546 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7gkh\" (UniqueName: \"kubernetes.io/projected/c0e0743c-3c3f-4283-936c-da9c4d2be4f3-kube-api-access-q7gkh\") pod \"maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx\" (UID: \"c0e0743c-3c3f-4283-936c-da9c4d2be4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 21:58:00.285790 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.285646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0e0743c-3c3f-4283-936c-da9c4d2be4f3-maas-api-tls\") pod \"maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx\" (UID: \"c0e0743c-3c3f-4283-936c-da9c4d2be4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 21:58:00.285790 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.285757 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0e0743c-3c3f-4283-936c-da9c4d2be4f3-openshift-service-ca\") pod \"maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx\" (UID: \"c0e0743c-3c3f-4283-936c-da9c4d2be4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 21:58:00.286454 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.286430 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0e0743c-3c3f-4283-936c-da9c4d2be4f3-openshift-service-ca\") pod \"maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx\" (UID: \"c0e0743c-3c3f-4283-936c-da9c4d2be4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 21:58:00.288344 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.288325 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0e0743c-3c3f-4283-936c-da9c4d2be4f3-maas-api-tls\") pod \"maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx\" (UID: \"c0e0743c-3c3f-4283-936c-da9c4d2be4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 21:58:00.299189 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.299163 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7gkh\" (UniqueName: \"kubernetes.io/projected/c0e0743c-3c3f-4283-936c-da9c4d2be4f3-kube-api-access-q7gkh\") pod \"maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx\" (UID: \"c0e0743c-3c3f-4283-936c-da9c4d2be4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 21:58:00.390876 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.390843 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:58:00.425910 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.425817 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 21:58:00.567070 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.567045 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx"] Jul 17 21:58:00.569104 ip-10-0-130-56 kubenswrapper[2580]: W0717 21:58:00.569081 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e0743c_3c3f_4283_936c_da9c4d2be4f3.slice/crio-179b32725595b7e81637960e645bc953b3e0e2be869912d0b2c0950353ea3452 WatchSource:0}: Error finding container 179b32725595b7e81637960e645bc953b3e0e2be869912d0b2c0950353ea3452: Status 404 returned error can't find the container with id 179b32725595b7e81637960e645bc953b3e0e2be869912d0b2c0950353ea3452 Jul 17 21:58:00.589875 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.589843 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" event={"ID":"c0e0743c-3c3f-4283-936c-da9c4d2be4f3","Type":"ContainerStarted","Data":"179b32725595b7e81637960e645bc953b3e0e2be869912d0b2c0950353ea3452"} Jul 17 21:58:00.684256 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.684181 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:58:00.785723 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:00.785686 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:58:01.596653 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:01.596556 2580 generic.go:356] "Generic (PLEG): container finished" podID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerID="a85175495156185993bbeb33b3b8638e569e5c0db37f543b150f92fbbe63cfc2" exitCode=1 Jul 17 21:58:01.597072 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:01.596643 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" event={"ID":"c0e0743c-3c3f-4283-936c-da9c4d2be4f3","Type":"ContainerDied","Data":"a85175495156185993bbeb33b3b8638e569e5c0db37f543b150f92fbbe63cfc2"} Jul 17 21:58:01.597072 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:01.596927 2580 scope.go:117] "RemoveContainer" containerID="a85175495156185993bbeb33b3b8638e569e5c0db37f543b150f92fbbe63cfc2" Jul 17 21:58:02.022858 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:02.022811 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:58:02.604827 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:02.604787 2580 generic.go:356] "Generic (PLEG): container finished" podID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerID="91152d0dff1fc950139132ef754ad1b10f2bc10d0daca858b5a853f62bdfdc3d" exitCode=1 Jul 17 21:58:02.605273 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:02.604855 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" event={"ID":"c0e0743c-3c3f-4283-936c-da9c4d2be4f3","Type":"ContainerDied","Data":"91152d0dff1fc950139132ef754ad1b10f2bc10d0daca858b5a853f62bdfdc3d"} Jul 17 21:58:02.605273 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:02.604897 2580 scope.go:117] "RemoveContainer" containerID="a85175495156185993bbeb33b3b8638e569e5c0db37f543b150f92fbbe63cfc2" Jul 17 21:58:02.605273 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:02.605089 2580 scope.go:117] "RemoveContainer" containerID="91152d0dff1fc950139132ef754ad1b10f2bc10d0daca858b5a853f62bdfdc3d" Jul 17 21:58:02.605444 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:58:02.605335 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 21:58:03.612838 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:03.612813 2580 scope.go:117] "RemoveContainer" containerID="91152d0dff1fc950139132ef754ad1b10f2bc10d0daca858b5a853f62bdfdc3d" Jul 17 21:58:03.613285 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:58:03.613048 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 21:58:05.426637 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:05.426600 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 21:58:05.427031 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:05.427002 2580 scope.go:117] "RemoveContainer" containerID="91152d0dff1fc950139132ef754ad1b10f2bc10d0daca858b5a853f62bdfdc3d" Jul 17 21:58:05.427187 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:58:05.427166 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 21:58:10.426912 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:10.426869 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 21:58:10.427426 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:10.427260 2580 scope.go:117] "RemoveContainer" containerID="91152d0dff1fc950139132ef754ad1b10f2bc10d0daca858b5a853f62bdfdc3d" Jul 17 21:58:10.427489 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:58:10.427469 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 21:58:25.018005 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:25.017971 2580 scope.go:117] "RemoveContainer" containerID="91152d0dff1fc950139132ef754ad1b10f2bc10d0daca858b5a853f62bdfdc3d" Jul 17 21:58:25.018430 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:25.018278 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 21:58:25.726017 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:25.725987 2580 generic.go:356] "Generic (PLEG): container finished" podID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerID="3a9fc4cafdfa93194c30b467aadc61f06f5a2d110530f8fe32bb2f68ed38212a" exitCode=1 Jul 17 21:58:25.726180 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:25.726066 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" event={"ID":"c0e0743c-3c3f-4283-936c-da9c4d2be4f3","Type":"ContainerDied","Data":"3a9fc4cafdfa93194c30b467aadc61f06f5a2d110530f8fe32bb2f68ed38212a"} Jul 17 21:58:25.726180 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:25.726130 2580 scope.go:117] "RemoveContainer" containerID="91152d0dff1fc950139132ef754ad1b10f2bc10d0daca858b5a853f62bdfdc3d" Jul 17 21:58:25.726472 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:25.726458 2580 scope.go:117] "RemoveContainer" containerID="3a9fc4cafdfa93194c30b467aadc61f06f5a2d110530f8fe32bb2f68ed38212a" Jul 17 21:58:25.726675 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:58:25.726658 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 21:58:30.426475 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:30.426427 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 21:58:30.426874 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:30.426504 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 21:58:30.426874 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:30.426827 2580 scope.go:117] "RemoveContainer" containerID="3a9fc4cafdfa93194c30b467aadc61f06f5a2d110530f8fe32bb2f68ed38212a" Jul 17 21:58:30.427022 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:58:30.427002 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 21:58:46.018381 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:46.018347 2580 scope.go:117] "RemoveContainer" containerID="3a9fc4cafdfa93194c30b467aadc61f06f5a2d110530f8fe32bb2f68ed38212a" Jul 17 21:58:46.846951 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:46.846913 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" event={"ID":"c0e0743c-3c3f-4283-936c-da9c4d2be4f3","Type":"ContainerStarted","Data":"90861e26607be95a96a7132b5bbf1982ea54fbdaeee7d5313791c110d16cc28f"} Jul 17 21:58:46.847148 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:46.847122 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 21:58:46.867022 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:46.866935 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podStartSLOduration=45.030692456 podStartE2EDuration="46.866918893s" totalImagesPullingTime="1.836226437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:58:00 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:58:46.864954196 +0000 UTC m=+1868.286673430" watchObservedRunningTime="2026-07-17 21:58:46.866918893 +0000 UTC m=+1868.288638104" Jul 17 21:58:47.853709 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:47.853670 2580 generic.go:356] "Generic (PLEG): container finished" podID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerID="90861e26607be95a96a7132b5bbf1982ea54fbdaeee7d5313791c110d16cc28f" exitCode=1 Jul 17 21:58:47.854206 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:47.853761 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" event={"ID":"c0e0743c-3c3f-4283-936c-da9c4d2be4f3","Type":"ContainerDied","Data":"90861e26607be95a96a7132b5bbf1982ea54fbdaeee7d5313791c110d16cc28f"} Jul 17 21:58:47.854206 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:47.853825 2580 scope.go:117] "RemoveContainer" containerID="3a9fc4cafdfa93194c30b467aadc61f06f5a2d110530f8fe32bb2f68ed38212a" Jul 17 21:58:47.854206 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:47.854042 2580 scope.go:117] "RemoveContainer" containerID="90861e26607be95a96a7132b5bbf1982ea54fbdaeee7d5313791c110d16cc28f" Jul 17 21:58:47.854371 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:58:47.854307 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 21:58:48.861129 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:48.861097 2580 scope.go:117] "RemoveContainer" containerID="90861e26607be95a96a7132b5bbf1982ea54fbdaeee7d5313791c110d16cc28f" Jul 17 21:58:48.861540 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:58:48.861286 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 21:58:50.426696 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:50.426661 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 21:58:50.427081 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:58:50.427017 2580 scope.go:117] "RemoveContainer" containerID="90861e26607be95a96a7132b5bbf1982ea54fbdaeee7d5313791c110d16cc28f" Jul 17 21:58:50.427215 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:58:50.427195 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 21:59:03.018513 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:59:03.018435 2580 scope.go:117] "RemoveContainer" containerID="90861e26607be95a96a7132b5bbf1982ea54fbdaeee7d5313791c110d16cc28f" Jul 17 21:59:03.018967 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:59:03.018640 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 21:59:17.018188 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:59:17.018149 2580 scope.go:117] "RemoveContainer" containerID="90861e26607be95a96a7132b5bbf1982ea54fbdaeee7d5313791c110d16cc28f" Jul 17 21:59:17.018777 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:59:17.018368 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 21:59:31.018647 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:59:31.018608 2580 scope.go:117] "RemoveContainer" containerID="90861e26607be95a96a7132b5bbf1982ea54fbdaeee7d5313791c110d16cc28f" Jul 17 21:59:32.085504 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:59:32.085473 2580 generic.go:356] "Generic (PLEG): container finished" podID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerID="5d1ba78f52f7daef38dea66dd7a6b870c98d5f5986b614922c4076eaf1512743" exitCode=1 Jul 17 21:59:32.085973 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:59:32.085542 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" event={"ID":"c0e0743c-3c3f-4283-936c-da9c4d2be4f3","Type":"ContainerDied","Data":"5d1ba78f52f7daef38dea66dd7a6b870c98d5f5986b614922c4076eaf1512743"} Jul 17 21:59:32.085973 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:59:32.085600 2580 scope.go:117] "RemoveContainer" containerID="90861e26607be95a96a7132b5bbf1982ea54fbdaeee7d5313791c110d16cc28f" Jul 17 21:59:32.086111 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:59:32.085984 2580 scope.go:117] "RemoveContainer" containerID="5d1ba78f52f7daef38dea66dd7a6b870c98d5f5986b614922c4076eaf1512743" Jul 17 21:59:32.086276 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:59:32.086250 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 21:59:36.847658 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:59:36.847622 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 21:59:36.848151 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:59:36.848058 2580 scope.go:117] "RemoveContainer" containerID="5d1ba78f52f7daef38dea66dd7a6b870c98d5f5986b614922c4076eaf1512743" Jul 17 21:59:36.848325 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:59:36.848301 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 21:59:40.426511 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:59:40.426471 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 21:59:40.426931 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:59:40.426880 2580 scope.go:117] "RemoveContainer" containerID="5d1ba78f52f7daef38dea66dd7a6b870c98d5f5986b614922c4076eaf1512743" Jul 17 21:59:40.427078 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:59:40.427057 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 21:59:40.599370 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:59:40.599324 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 21:59:55.018338 ip-10-0-130-56 kubenswrapper[2580]: I0717 21:59:55.018303 2580 scope.go:117] "RemoveContainer" containerID="5d1ba78f52f7daef38dea66dd7a6b870c98d5f5986b614922c4076eaf1512743" Jul 17 21:59:55.018867 ip-10-0-130-56 kubenswrapper[2580]: E0717 21:59:55.018519 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:00:10.018570 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:00:10.018538 2580 scope.go:117] "RemoveContainer" containerID="5d1ba78f52f7daef38dea66dd7a6b870c98d5f5986b614922c4076eaf1512743" Jul 17 22:00:10.018980 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:00:10.018726 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:00:21.023804 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:00:21.023766 2580 scope.go:117] "RemoveContainer" containerID="5d1ba78f52f7daef38dea66dd7a6b870c98d5f5986b614922c4076eaf1512743" Jul 17 22:00:21.024242 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:00:21.023974 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:00:35.024198 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:00:35.024122 2580 scope.go:117] "RemoveContainer" containerID="5d1ba78f52f7daef38dea66dd7a6b870c98d5f5986b614922c4076eaf1512743" Jul 17 22:00:35.024701 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:00:35.024337 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:00:50.017973 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:00:50.017934 2580 scope.go:117] "RemoveContainer" containerID="5d1ba78f52f7daef38dea66dd7a6b870c98d5f5986b614922c4076eaf1512743" Jul 17 22:00:50.018425 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:00:50.018168 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:01:03.018253 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:01:03.018213 2580 scope.go:117] "RemoveContainer" containerID="5d1ba78f52f7daef38dea66dd7a6b870c98d5f5986b614922c4076eaf1512743" Jul 17 22:01:03.573259 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:01:03.573222 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" event={"ID":"c0e0743c-3c3f-4283-936c-da9c4d2be4f3","Type":"ContainerStarted","Data":"5202d0add5ea26af9540bde0590f40b2fbed66677e99e463d8d214ceaa15568b"} Jul 17 22:01:03.573456 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:01:03.573443 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 22:01:04.580682 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:01:04.580650 2580 generic.go:356] "Generic (PLEG): container finished" podID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerID="5202d0add5ea26af9540bde0590f40b2fbed66677e99e463d8d214ceaa15568b" exitCode=1 Jul 17 22:01:04.581121 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:01:04.580737 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" event={"ID":"c0e0743c-3c3f-4283-936c-da9c4d2be4f3","Type":"ContainerDied","Data":"5202d0add5ea26af9540bde0590f40b2fbed66677e99e463d8d214ceaa15568b"} Jul 17 22:01:04.581121 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:01:04.580799 2580 scope.go:117] "RemoveContainer" containerID="5d1ba78f52f7daef38dea66dd7a6b870c98d5f5986b614922c4076eaf1512743" Jul 17 22:01:04.581121 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:01:04.581018 2580 scope.go:117] "RemoveContainer" containerID="5202d0add5ea26af9540bde0590f40b2fbed66677e99e463d8d214ceaa15568b" Jul 17 22:01:04.581265 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:01:04.581238 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:01:05.587602 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:01:05.587577 2580 scope.go:117] "RemoveContainer" containerID="5202d0add5ea26af9540bde0590f40b2fbed66677e99e463d8d214ceaa15568b" Jul 17 22:01:05.588048 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:01:05.587775 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:01:10.425964 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:01:10.425924 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 22:01:10.426477 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:01:10.426361 2580 scope.go:117] "RemoveContainer" containerID="5202d0add5ea26af9540bde0590f40b2fbed66677e99e463d8d214ceaa15568b" Jul 17 22:01:10.426622 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:01:10.426599 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:01:24.018670 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:01:24.018637 2580 scope.go:117] "RemoveContainer" containerID="5202d0add5ea26af9540bde0590f40b2fbed66677e99e463d8d214ceaa15568b" Jul 17 22:01:24.019060 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:01:24.018820 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:01:28.059887 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:01:28.059854 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:01:31.592690 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:01:31.592645 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:01:31.692917 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:01:31.692883 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:01:39.020735 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:01:39.020699 2580 scope.go:117] "RemoveContainer" containerID="5202d0add5ea26af9540bde0590f40b2fbed66677e99e463d8d214ceaa15568b" Jul 17 22:01:39.021257 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:01:39.020915 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:01:45.719973 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:01:45.719938 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:01:45.826616 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:01:45.826579 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:01:51.256521 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:01:51.256487 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:01:54.018831 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:01:54.018796 2580 scope.go:117] "RemoveContainer" containerID="5202d0add5ea26af9540bde0590f40b2fbed66677e99e463d8d214ceaa15568b" Jul 17 22:01:54.019234 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:01:54.019007 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:02:09.021078 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:02:09.021049 2580 scope.go:117] "RemoveContainer" containerID="5202d0add5ea26af9540bde0590f40b2fbed66677e99e463d8d214ceaa15568b" Jul 17 22:02:09.021512 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:02:09.021238 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:02:21.018873 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:02:21.018837 2580 scope.go:117] "RemoveContainer" containerID="5202d0add5ea26af9540bde0590f40b2fbed66677e99e463d8d214ceaa15568b" Jul 17 22:02:21.019326 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:02:21.019074 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:02:36.018818 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:02:36.018781 2580 scope.go:117] "RemoveContainer" containerID="5202d0add5ea26af9540bde0590f40b2fbed66677e99e463d8d214ceaa15568b" Jul 17 22:02:36.019206 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:02:36.018964 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:02:39.140465 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:02:39.140436 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 22:02:39.144806 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:02:39.144785 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 22:02:49.022828 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:02:49.022796 2580 scope.go:117] "RemoveContainer" containerID="5202d0add5ea26af9540bde0590f40b2fbed66677e99e463d8d214ceaa15568b" Jul 17 22:02:49.023212 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:02:49.023060 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:03:01.018979 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:03:01.018949 2580 scope.go:117] "RemoveContainer" containerID="5202d0add5ea26af9540bde0590f40b2fbed66677e99e463d8d214ceaa15568b" Jul 17 22:03:01.019471 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:03:01.019128 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:03:13.018055 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:03:13.018015 2580 scope.go:117] "RemoveContainer" containerID="5202d0add5ea26af9540bde0590f40b2fbed66677e99e463d8d214ceaa15568b" Jul 17 22:03:13.018577 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:03:13.018272 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:03:24.018731 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:03:24.018696 2580 scope.go:117] "RemoveContainer" containerID="5202d0add5ea26af9540bde0590f40b2fbed66677e99e463d8d214ceaa15568b" Jul 17 22:03:24.019201 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:03:24.018876 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:03:37.018511 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:03:37.018422 2580 scope.go:117] "RemoveContainer" containerID="5202d0add5ea26af9540bde0590f40b2fbed66677e99e463d8d214ceaa15568b" Jul 17 22:03:37.018917 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:03:37.018591 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:03:51.018048 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:03:51.018018 2580 scope.go:117] "RemoveContainer" containerID="5202d0add5ea26af9540bde0590f40b2fbed66677e99e463d8d214ceaa15568b" Jul 17 22:03:51.018492 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:03:51.018317 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 22:03:52.483107 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:03:52.483075 2580 generic.go:356] "Generic (PLEG): container finished" podID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerID="896bfadcf862eb33a7b8e0a6799b2446bf0d7514f7425c5d4414cc61dad12311" exitCode=1 Jul 17 22:03:52.483564 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:03:52.483151 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" event={"ID":"c0e0743c-3c3f-4283-936c-da9c4d2be4f3","Type":"ContainerDied","Data":"896bfadcf862eb33a7b8e0a6799b2446bf0d7514f7425c5d4414cc61dad12311"} Jul 17 22:03:52.483564 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:03:52.483214 2580 scope.go:117] "RemoveContainer" containerID="5202d0add5ea26af9540bde0590f40b2fbed66677e99e463d8d214ceaa15568b" Jul 17 22:03:52.483683 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:03:52.483648 2580 scope.go:117] "RemoveContainer" containerID="896bfadcf862eb33a7b8e0a6799b2446bf0d7514f7425c5d4414cc61dad12311" Jul 17 22:03:52.483878 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:03:52.483849 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:03:53.574309 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:03:53.574279 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 22:03:53.574739 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:03:53.574707 2580 scope.go:117] "RemoveContainer" containerID="896bfadcf862eb33a7b8e0a6799b2446bf0d7514f7425c5d4414cc61dad12311" Jul 17 22:03:53.574901 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:03:53.574880 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:04:00.426779 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:00.426737 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 22:04:00.427242 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:00.427131 2580 scope.go:117] "RemoveContainer" containerID="896bfadcf862eb33a7b8e0a6799b2446bf0d7514f7425c5d4414cc61dad12311" Jul 17 22:04:00.427362 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:04:00.427337 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx_odh-ai-gateway-infra(c0e0743c-3c3f-4283-936c-da9c4d2be4f3)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" Jul 17 22:04:05.295755 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:05.295713 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:04:06.502516 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:06.502483 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:04:10.481174 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:10.481137 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx"] Jul 17 22:04:10.505844 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:10.505813 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:04:10.625486 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:10.625465 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 22:04:10.690715 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:10.690687 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0e0743c-3c3f-4283-936c-da9c4d2be4f3-maas-api-tls\") pod \"c0e0743c-3c3f-4283-936c-da9c4d2be4f3\" (UID: \"c0e0743c-3c3f-4283-936c-da9c4d2be4f3\") " Jul 17 22:04:10.690869 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:10.690724 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7gkh\" (UniqueName: \"kubernetes.io/projected/c0e0743c-3c3f-4283-936c-da9c4d2be4f3-kube-api-access-q7gkh\") pod \"c0e0743c-3c3f-4283-936c-da9c4d2be4f3\" (UID: \"c0e0743c-3c3f-4283-936c-da9c4d2be4f3\") " Jul 17 22:04:10.690869 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:10.690789 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0e0743c-3c3f-4283-936c-da9c4d2be4f3-openshift-service-ca\") pod \"c0e0743c-3c3f-4283-936c-da9c4d2be4f3\" (UID: \"c0e0743c-3c3f-4283-936c-da9c4d2be4f3\") " Jul 17 22:04:10.691182 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:10.691156 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e0743c-3c3f-4283-936c-da9c4d2be4f3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c0e0743c-3c3f-4283-936c-da9c4d2be4f3" (UID: "c0e0743c-3c3f-4283-936c-da9c4d2be4f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:04:10.692964 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:10.692939 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e0743c-3c3f-4283-936c-da9c4d2be4f3-kube-api-access-q7gkh" (OuterVolumeSpecName: "kube-api-access-q7gkh") pod "c0e0743c-3c3f-4283-936c-da9c4d2be4f3" (UID: "c0e0743c-3c3f-4283-936c-da9c4d2be4f3"). InnerVolumeSpecName "kube-api-access-q7gkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:04:10.693067 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:10.692979 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0e0743c-3c3f-4283-936c-da9c4d2be4f3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c0e0743c-3c3f-4283-936c-da9c4d2be4f3" (UID: "c0e0743c-3c3f-4283-936c-da9c4d2be4f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:04:10.791588 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:10.791556 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0e0743c-3c3f-4283-936c-da9c4d2be4f3-openshift-service-ca\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:04:10.791588 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:10.791585 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0e0743c-3c3f-4283-936c-da9c4d2be4f3-maas-api-tls\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:04:10.791588 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:10.791596 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7gkh\" (UniqueName: \"kubernetes.io/projected/c0e0743c-3c3f-4283-936c-da9c4d2be4f3-kube-api-access-q7gkh\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:04:11.280825 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:11.280797 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:04:11.587458 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:11.587352 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" event={"ID":"c0e0743c-3c3f-4283-936c-da9c4d2be4f3","Type":"ContainerDied","Data":"179b32725595b7e81637960e645bc953b3e0e2be869912d0b2c0950353ea3452"} Jul 17 22:04:11.587458 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:11.587400 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx" Jul 17 22:04:11.587458 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:11.587430 2580 scope.go:117] "RemoveContainer" containerID="896bfadcf862eb33a7b8e0a6799b2446bf0d7514f7425c5d4414cc61dad12311" Jul 17 22:04:11.604767 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:11.604743 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx"] Jul 17 22:04:11.615659 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:11.615631 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e3cf3d26-759b6f6c65-r7jcx"] Jul 17 22:04:12.345188 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:12.345126 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:04:13.023544 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:13.023511 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" path="/var/lib/kubelet/pods/c0e0743c-3c3f-4283-936c-da9c4d2be4f3/volumes" Jul 17 22:04:16.976526 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:16.976490 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:04:17.052122 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:17.052083 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:04:17.268021 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:17.267988 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:04:17.367503 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:17.367471 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:04:18.399330 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.399280 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:04:18.504078 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504044 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg"] Jul 17 22:04:18.504573 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504552 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:04:18.504573 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504571 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:04:18.504736 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504584 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" containerName="maas-api" Jul 17 22:04:18.504736 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504590 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" containerName="maas-api" Jul 17 22:04:18.504736 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504596 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:04:18.504736 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504601 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:04:18.504736 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504609 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:04:18.504736 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504616 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:04:18.504736 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504628 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:04:18.504736 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504635 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:04:18.504736 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504641 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:04:18.504736 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504646 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:04:18.504736 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504720 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a28a25a4-e7d2-42da-bb32-ba5495582af8" containerName="maas-api" Jul 17 22:04:18.504736 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504730 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:04:18.504736 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504737 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:04:18.504736 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504743 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:04:18.505489 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504749 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:04:18.505489 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504755 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c84a2810-3320-4044-ba33-3ae9df132f96" containerName="maas-api" Jul 17 22:04:18.505489 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.504760 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:04:18.507448 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.507384 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:04:18.508034 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.508016 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:04:18.510555 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.510535 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-6f4935d4-serving-cert\"" Jul 17 22:04:18.520510 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.520484 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg"] Jul 17 22:04:18.597582 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.597544 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:04:18.666101 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.666011 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fbf3d87-47ce-4a7b-b944-6f02a23d31dd-openshift-service-ca\") pod \"maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg\" (UID: \"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:04:18.666281 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.666215 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x975g\" (UniqueName: \"kubernetes.io/projected/8fbf3d87-47ce-4a7b-b944-6f02a23d31dd-kube-api-access-x975g\") pod \"maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg\" (UID: \"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:04:18.666353 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.666312 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fbf3d87-47ce-4a7b-b944-6f02a23d31dd-maas-api-tls\") pod \"maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg\" (UID: \"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:04:18.767321 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.767284 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fbf3d87-47ce-4a7b-b944-6f02a23d31dd-openshift-service-ca\") pod \"maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg\" (UID: \"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:04:18.767321 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.767322 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x975g\" (UniqueName: \"kubernetes.io/projected/8fbf3d87-47ce-4a7b-b944-6f02a23d31dd-kube-api-access-x975g\") pod \"maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg\" (UID: \"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:04:18.767841 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.767372 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fbf3d87-47ce-4a7b-b944-6f02a23d31dd-maas-api-tls\") pod \"maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg\" (UID: \"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:04:18.767940 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.767920 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fbf3d87-47ce-4a7b-b944-6f02a23d31dd-openshift-service-ca\") pod \"maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg\" (UID: \"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:04:18.770145 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.770110 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fbf3d87-47ce-4a7b-b944-6f02a23d31dd-maas-api-tls\") pod \"maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg\" (UID: \"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:04:18.775783 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.775763 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x975g\" (UniqueName: \"kubernetes.io/projected/8fbf3d87-47ce-4a7b-b944-6f02a23d31dd-kube-api-access-x975g\") pod \"maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg\" (UID: \"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:04:18.803436 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.803406 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:04:18.821990 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.821967 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:04:18.961647 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:18.961621 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg"] Jul 17 22:04:18.963594 ip-10-0-130-56 kubenswrapper[2580]: W0717 22:04:18.963568 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbf3d87_47ce_4a7b_b944_6f02a23d31dd.slice/crio-2a6910dc04ce82e5b0ad0d4117b5a15c04f9dce3437d28ef4a46c125a920a845 WatchSource:0}: Error finding container 2a6910dc04ce82e5b0ad0d4117b5a15c04f9dce3437d28ef4a46c125a920a845: Status 404 returned error can't find the container with id 2a6910dc04ce82e5b0ad0d4117b5a15c04f9dce3437d28ef4a46c125a920a845 Jul 17 22:04:19.163681 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:19.163650 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:04:19.249990 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:19.249902 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:04:19.633455 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:19.633418 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" event={"ID":"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd","Type":"ContainerStarted","Data":"e7a5394e4ceefe919184dfc0e7a40dd8704def3d27084516b2105ff5f5738dc9"} Jul 17 22:04:19.633849 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:19.633485 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" event={"ID":"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd","Type":"ContainerStarted","Data":"2a6910dc04ce82e5b0ad0d4117b5a15c04f9dce3437d28ef4a46c125a920a845"} Jul 17 22:04:19.633849 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:19.633541 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:04:19.649581 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:19.649535 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podStartSLOduration=1.254015034 podStartE2EDuration="1.649521185s" totalImagesPullingTime="395.506151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:04:19.648538782 +0000 UTC m=+2201.070258014" watchObservedRunningTime="2026-07-17 22:04:19.649521185 +0000 UTC m=+2201.071240417" Jul 17 22:04:20.641475 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:20.641438 2580 generic.go:356] "Generic (PLEG): container finished" podID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerID="e7a5394e4ceefe919184dfc0e7a40dd8704def3d27084516b2105ff5f5738dc9" exitCode=1 Jul 17 22:04:20.641894 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:20.641483 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" event={"ID":"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd","Type":"ContainerDied","Data":"e7a5394e4ceefe919184dfc0e7a40dd8704def3d27084516b2105ff5f5738dc9"} Jul 17 22:04:20.641894 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:20.641809 2580 scope.go:117] "RemoveContainer" containerID="e7a5394e4ceefe919184dfc0e7a40dd8704def3d27084516b2105ff5f5738dc9" Jul 17 22:04:21.530295 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:21.530265 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:04:21.648639 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:21.648544 2580 generic.go:356] "Generic (PLEG): container finished" podID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerID="df985906cb0dcdbda1ed94db5e7d5a04dcabca160d7e9482ff7a9a95b91a2b57" exitCode=1 Jul 17 22:04:21.649016 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:21.648634 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" event={"ID":"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd","Type":"ContainerDied","Data":"df985906cb0dcdbda1ed94db5e7d5a04dcabca160d7e9482ff7a9a95b91a2b57"} Jul 17 22:04:21.649016 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:21.648700 2580 scope.go:117] "RemoveContainer" containerID="e7a5394e4ceefe919184dfc0e7a40dd8704def3d27084516b2105ff5f5738dc9" Jul 17 22:04:21.649016 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:21.648931 2580 scope.go:117] "RemoveContainer" containerID="df985906cb0dcdbda1ed94db5e7d5a04dcabca160d7e9482ff7a9a95b91a2b57" Jul 17 22:04:21.649147 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:04:21.649124 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:04:22.656169 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:22.656105 2580 scope.go:117] "RemoveContainer" containerID="df985906cb0dcdbda1ed94db5e7d5a04dcabca160d7e9482ff7a9a95b91a2b57" Jul 17 22:04:22.656595 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:04:22.656296 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:04:24.634302 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:24.634251 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:04:24.634730 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:24.634656 2580 scope.go:117] "RemoveContainer" containerID="df985906cb0dcdbda1ed94db5e7d5a04dcabca160d7e9482ff7a9a95b91a2b57" Jul 17 22:04:24.634844 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:04:24.634825 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:04:28.822870 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:28.822825 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:04:28.823365 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:28.823247 2580 scope.go:117] "RemoveContainer" containerID="df985906cb0dcdbda1ed94db5e7d5a04dcabca160d7e9482ff7a9a95b91a2b57" Jul 17 22:04:28.823471 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:04:28.823448 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:04:42.017812 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:42.017779 2580 scope.go:117] "RemoveContainer" containerID="df985906cb0dcdbda1ed94db5e7d5a04dcabca160d7e9482ff7a9a95b91a2b57" Jul 17 22:04:42.764891 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:42.764855 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" event={"ID":"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd","Type":"ContainerStarted","Data":"e5b8519ab51fa1c1dea5470afd2f0d1fa6c76ce4d0dec2c5da8e4288b829a147"} Jul 17 22:04:42.765099 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:42.765075 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:04:43.772366 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:43.772327 2580 generic.go:356] "Generic (PLEG): container finished" podID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerID="e5b8519ab51fa1c1dea5470afd2f0d1fa6c76ce4d0dec2c5da8e4288b829a147" exitCode=1 Jul 17 22:04:43.772882 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:43.772418 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" event={"ID":"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd","Type":"ContainerDied","Data":"e5b8519ab51fa1c1dea5470afd2f0d1fa6c76ce4d0dec2c5da8e4288b829a147"} Jul 17 22:04:43.772882 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:43.772479 2580 scope.go:117] "RemoveContainer" containerID="df985906cb0dcdbda1ed94db5e7d5a04dcabca160d7e9482ff7a9a95b91a2b57" Jul 17 22:04:43.772882 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:43.772690 2580 scope.go:117] "RemoveContainer" containerID="e5b8519ab51fa1c1dea5470afd2f0d1fa6c76ce4d0dec2c5da8e4288b829a147" Jul 17 22:04:43.773003 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:04:43.772916 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:04:44.779578 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:44.779553 2580 scope.go:117] "RemoveContainer" containerID="e5b8519ab51fa1c1dea5470afd2f0d1fa6c76ce4d0dec2c5da8e4288b829a147" Jul 17 22:04:44.779956 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:04:44.779738 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:04:48.822474 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:48.822444 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:04:48.822875 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:04:48.822827 2580 scope.go:117] "RemoveContainer" containerID="e5b8519ab51fa1c1dea5470afd2f0d1fa6c76ce4d0dec2c5da8e4288b829a147" Jul 17 22:04:48.823014 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:04:48.822991 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:05:03.018370 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:03.018295 2580 scope.go:117] "RemoveContainer" containerID="e5b8519ab51fa1c1dea5470afd2f0d1fa6c76ce4d0dec2c5da8e4288b829a147" Jul 17 22:05:03.880824 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:03.880790 2580 generic.go:356] "Generic (PLEG): container finished" podID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerID="71b38927eb97a49f3f76c435445a8b2d439c5dcb9fdfa4a814886f1ca13c2062" exitCode=1 Jul 17 22:05:03.881000 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:03.880866 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" event={"ID":"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd","Type":"ContainerDied","Data":"71b38927eb97a49f3f76c435445a8b2d439c5dcb9fdfa4a814886f1ca13c2062"} Jul 17 22:05:03.881000 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:03.880930 2580 scope.go:117] "RemoveContainer" containerID="e5b8519ab51fa1c1dea5470afd2f0d1fa6c76ce4d0dec2c5da8e4288b829a147" Jul 17 22:05:03.881336 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:03.881320 2580 scope.go:117] "RemoveContainer" containerID="71b38927eb97a49f3f76c435445a8b2d439c5dcb9fdfa4a814886f1ca13c2062" Jul 17 22:05:03.881587 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:05:03.881568 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:05:07.765364 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:07.765323 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:05:07.765813 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:07.765794 2580 scope.go:117] "RemoveContainer" containerID="71b38927eb97a49f3f76c435445a8b2d439c5dcb9fdfa4a814886f1ca13c2062" Jul 17 22:05:07.766071 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:05:07.766049 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:05:08.822119 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:08.822081 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:05:08.822586 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:08.822464 2580 scope.go:117] "RemoveContainer" containerID="71b38927eb97a49f3f76c435445a8b2d439c5dcb9fdfa4a814886f1ca13c2062" Jul 17 22:05:08.822662 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:05:08.822641 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:05:11.388933 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:11.388897 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:05:11.449464 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:11.449433 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:05:21.018754 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:21.018719 2580 scope.go:117] "RemoveContainer" containerID="71b38927eb97a49f3f76c435445a8b2d439c5dcb9fdfa4a814886f1ca13c2062" Jul 17 22:05:21.019220 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:05:21.018893 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:05:36.018479 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:36.018448 2580 scope.go:117] "RemoveContainer" containerID="71b38927eb97a49f3f76c435445a8b2d439c5dcb9fdfa4a814886f1ca13c2062" Jul 17 22:05:36.018989 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:05:36.018665 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:05:49.161219 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:49.161182 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:05:49.677714 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:49.677679 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:05:50.018551 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:50.018519 2580 scope.go:117] "RemoveContainer" containerID="71b38927eb97a49f3f76c435445a8b2d439c5dcb9fdfa4a814886f1ca13c2062" Jul 17 22:05:51.135647 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:51.135617 2580 generic.go:356] "Generic (PLEG): container finished" podID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerID="8b9d55e4b9aae4c67aaa746dd899b40b0fc1db3c699da469ded2bfe26ba2c865" exitCode=1 Jul 17 22:05:51.136073 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:51.135686 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" event={"ID":"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd","Type":"ContainerDied","Data":"8b9d55e4b9aae4c67aaa746dd899b40b0fc1db3c699da469ded2bfe26ba2c865"} Jul 17 22:05:51.136073 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:51.135753 2580 scope.go:117] "RemoveContainer" containerID="71b38927eb97a49f3f76c435445a8b2d439c5dcb9fdfa4a814886f1ca13c2062" Jul 17 22:05:51.136195 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:51.136126 2580 scope.go:117] "RemoveContainer" containerID="8b9d55e4b9aae4c67aaa746dd899b40b0fc1db3c699da469ded2bfe26ba2c865" Jul 17 22:05:51.136357 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:05:51.136337 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:05:52.766056 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:52.766012 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:05:52.766545 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:52.766434 2580 scope.go:117] "RemoveContainer" containerID="8b9d55e4b9aae4c67aaa746dd899b40b0fc1db3c699da469ded2bfe26ba2c865" Jul 17 22:05:52.766648 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:05:52.766628 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:05:58.822652 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:58.822611 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:05:58.823106 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:05:58.823089 2580 scope.go:117] "RemoveContainer" containerID="8b9d55e4b9aae4c67aaa746dd899b40b0fc1db3c699da469ded2bfe26ba2c865" Jul 17 22:05:58.823320 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:05:58.823299 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:06:11.017932 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:06:11.017899 2580 scope.go:117] "RemoveContainer" containerID="8b9d55e4b9aae4c67aaa746dd899b40b0fc1db3c699da469ded2bfe26ba2c865" Jul 17 22:06:11.018469 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:06:11.018100 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:06:26.018711 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:06:26.018676 2580 scope.go:117] "RemoveContainer" containerID="8b9d55e4b9aae4c67aaa746dd899b40b0fc1db3c699da469ded2bfe26ba2c865" Jul 17 22:06:26.019173 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:06:26.018862 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:06:32.419291 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:06:32.419166 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:06:32.520208 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:06:32.520163 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:06:35.058406 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:06:35.058355 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:06:35.159377 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:06:35.159334 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:06:39.023296 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:06:39.023256 2580 scope.go:117] "RemoveContainer" containerID="8b9d55e4b9aae4c67aaa746dd899b40b0fc1db3c699da469ded2bfe26ba2c865" Jul 17 22:06:39.024196 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:06:39.024161 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:06:51.018435 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:06:51.018366 2580 scope.go:117] "RemoveContainer" containerID="8b9d55e4b9aae4c67aaa746dd899b40b0fc1db3c699da469ded2bfe26ba2c865" Jul 17 22:06:51.018840 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:06:51.018664 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:07:03.018511 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:03.018478 2580 scope.go:117] "RemoveContainer" containerID="8b9d55e4b9aae4c67aaa746dd899b40b0fc1db3c699da469ded2bfe26ba2c865" Jul 17 22:07:03.018912 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:07:03.018671 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:07:14.018463 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:14.018427 2580 scope.go:117] "RemoveContainer" containerID="8b9d55e4b9aae4c67aaa746dd899b40b0fc1db3c699da469ded2bfe26ba2c865" Jul 17 22:07:14.570714 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:14.570678 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" event={"ID":"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd","Type":"ContainerStarted","Data":"271439a6522ac56d4d3ecb9f7cc747431f81e8c8bf90f41815f56f88c060598b"} Jul 17 22:07:14.570918 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:14.570900 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:07:15.578571 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:15.578534 2580 generic.go:356] "Generic (PLEG): container finished" podID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerID="271439a6522ac56d4d3ecb9f7cc747431f81e8c8bf90f41815f56f88c060598b" exitCode=1 Jul 17 22:07:15.579049 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:15.578623 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" event={"ID":"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd","Type":"ContainerDied","Data":"271439a6522ac56d4d3ecb9f7cc747431f81e8c8bf90f41815f56f88c060598b"} Jul 17 22:07:15.579049 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:15.578683 2580 scope.go:117] "RemoveContainer" containerID="8b9d55e4b9aae4c67aaa746dd899b40b0fc1db3c699da469ded2bfe26ba2c865" Jul 17 22:07:15.579049 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:15.578922 2580 scope.go:117] "RemoveContainer" containerID="271439a6522ac56d4d3ecb9f7cc747431f81e8c8bf90f41815f56f88c060598b" Jul 17 22:07:15.579181 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:07:15.579154 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:07:16.585719 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:16.585694 2580 scope.go:117] "RemoveContainer" containerID="271439a6522ac56d4d3ecb9f7cc747431f81e8c8bf90f41815f56f88c060598b" Jul 17 22:07:16.586206 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:07:16.585879 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:07:18.822782 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:18.822751 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:07:18.823298 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:18.823241 2580 scope.go:117] "RemoveContainer" containerID="271439a6522ac56d4d3ecb9f7cc747431f81e8c8bf90f41815f56f88c060598b" Jul 17 22:07:18.823507 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:07:18.823483 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg_odh-ai-gateway-infra(8fbf3d87-47ce-4a7b-b944-6f02a23d31dd)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" Jul 17 22:07:22.956661 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:22.956624 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:07:23.056433 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:23.056385 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:07:28.432905 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:28.432867 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg"] Jul 17 22:07:28.475803 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:28.475776 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:07:28.567523 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:28.567499 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:07:28.651574 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:28.651545 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" Jul 17 22:07:28.651745 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:28.651581 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg" event={"ID":"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd","Type":"ContainerDied","Data":"2a6910dc04ce82e5b0ad0d4117b5a15c04f9dce3437d28ef4a46c125a920a845"} Jul 17 22:07:28.651745 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:28.651641 2580 scope.go:117] "RemoveContainer" containerID="271439a6522ac56d4d3ecb9f7cc747431f81e8c8bf90f41815f56f88c060598b" Jul 17 22:07:28.684808 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:28.684779 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fbf3d87-47ce-4a7b-b944-6f02a23d31dd-maas-api-tls\") pod \"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd\" (UID: \"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd\") " Jul 17 22:07:28.684957 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:28.684892 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fbf3d87-47ce-4a7b-b944-6f02a23d31dd-openshift-service-ca\") pod \"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd\" (UID: \"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd\") " Jul 17 22:07:28.684957 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:28.684926 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x975g\" (UniqueName: \"kubernetes.io/projected/8fbf3d87-47ce-4a7b-b944-6f02a23d31dd-kube-api-access-x975g\") pod \"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd\" (UID: \"8fbf3d87-47ce-4a7b-b944-6f02a23d31dd\") " Jul 17 22:07:28.685216 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:28.685189 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fbf3d87-47ce-4a7b-b944-6f02a23d31dd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" (UID: "8fbf3d87-47ce-4a7b-b944-6f02a23d31dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:07:28.687058 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:28.687036 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fbf3d87-47ce-4a7b-b944-6f02a23d31dd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" (UID: "8fbf3d87-47ce-4a7b-b944-6f02a23d31dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:07:28.687144 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:28.687067 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbf3d87-47ce-4a7b-b944-6f02a23d31dd-kube-api-access-x975g" (OuterVolumeSpecName: "kube-api-access-x975g") pod "8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" (UID: "8fbf3d87-47ce-4a7b-b944-6f02a23d31dd"). InnerVolumeSpecName "kube-api-access-x975g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:07:28.786314 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:28.786279 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fbf3d87-47ce-4a7b-b944-6f02a23d31dd-maas-api-tls\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:07:28.786314 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:28.786311 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fbf3d87-47ce-4a7b-b944-6f02a23d31dd-openshift-service-ca\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:07:28.786553 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:28.786323 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x975g\" (UniqueName: \"kubernetes.io/projected/8fbf3d87-47ce-4a7b-b944-6f02a23d31dd-kube-api-access-x975g\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:07:28.974348 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:28.974317 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg"] Jul 17 22:07:28.977785 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:28.977761 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-6f4935d4-6b75c9bc5f-8rzsg"] Jul 17 22:07:29.025291 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:29.025261 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" path="/var/lib/kubelet/pods/8fbf3d87-47ce-4a7b-b944-6f02a23d31dd/volumes" Jul 17 22:07:29.147277 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:29.147188 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:07:34.797590 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:34.797556 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:07:34.900011 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:34.899979 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:07:35.279262 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:35.279226 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:07:35.378356 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:35.378324 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:07:35.479254 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:35.479201 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:07:36.950682 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:36.950649 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:07:39.174124 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:39.174093 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 22:07:39.180029 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:39.180009 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 22:07:39.490913 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:39.490874 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:07:42.941223 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:42.941187 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:07:43.050913 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:43.050879 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:07:43.147174 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:43.147136 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:07:43.248968 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:43.248890 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:07:43.662224 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:43.662187 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:07:54.829648 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:07:54.829612 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:08:09.635845 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:08:09.635790 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:08:15.819313 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:08:15.819276 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:08:16.058705 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:08:16.058674 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:08:16.219635 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:08:16.219556 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:08:32.028662 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:08:32.028624 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:08:37.882166 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:08:37.882131 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:08:37.936896 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:08:37.936860 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:08:38.006906 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:08:38.006869 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:08:38.106578 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:08:38.106546 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:08:48.611445 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:08:48.611409 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:08:49.432037 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:08:49.432001 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:08:55.153744 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:08:55.153705 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:08:55.256870 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:08:55.256830 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:08:55.354270 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:08:55.354236 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:08:55.454438 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:08:55.454333 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:08:56.246455 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:08:56.246422 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:08:56.338162 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:08:56.338130 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:08:56.439427 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:08:56.439370 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:08:58.049408 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:08:58.049370 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:03.790375 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:03.790322 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:03.854125 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:03.854090 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:04.467891 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:04.467858 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:10.858309 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:10.858268 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:12.243684 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:12.243648 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:14.119289 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.119254 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:14.202187 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.202149 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:14.208119 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208082 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7"] Jul 17 22:09:14.208590 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208570 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerName="maas-api" Jul 17 22:09:14.208590 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208591 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerName="maas-api" Jul 17 22:09:14.208739 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208605 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerName="maas-api" Jul 17 22:09:14.208739 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208611 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerName="maas-api" Jul 17 22:09:14.208739 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208623 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerName="maas-api" Jul 17 22:09:14.208739 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208628 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerName="maas-api" Jul 17 22:09:14.208739 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208635 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerName="maas-api" Jul 17 22:09:14.208739 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208640 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerName="maas-api" Jul 17 22:09:14.208739 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208653 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:09:14.208739 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208657 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:09:14.208739 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208664 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:09:14.208739 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208669 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:09:14.208739 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208722 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerName="maas-api" Jul 17 22:09:14.208739 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208730 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerName="maas-api" Jul 17 22:09:14.208739 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208737 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerName="maas-api" Jul 17 22:09:14.208739 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208743 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:09:14.208739 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208749 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0e0743c-3c3f-4283-936c-da9c4d2be4f3" containerName="maas-api" Jul 17 22:09:14.209317 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208754 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerName="maas-api" Jul 17 22:09:14.209317 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208760 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerName="maas-api" Jul 17 22:09:14.209317 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.208766 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerName="maas-api" Jul 17 22:09:14.212087 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.212059 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" Jul 17 22:09:14.214878 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.214861 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8962d973-serving-cert\"" Jul 17 22:09:14.223411 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.223351 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7"] Jul 17 22:09:14.241456 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.241428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd039b73-b041-405b-9124-827ad244009a-openshift-service-ca\") pod \"maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7\" (UID: \"cd039b73-b041-405b-9124-827ad244009a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" Jul 17 22:09:14.241591 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.241467 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zfv9\" (UniqueName: \"kubernetes.io/projected/cd039b73-b041-405b-9124-827ad244009a-kube-api-access-2zfv9\") pod \"maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7\" (UID: \"cd039b73-b041-405b-9124-827ad244009a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" Jul 17 22:09:14.241591 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.241567 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd039b73-b041-405b-9124-827ad244009a-maas-api-tls\") pod \"maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7\" (UID: \"cd039b73-b041-405b-9124-827ad244009a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" Jul 17 22:09:14.266676 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.266638 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:14.342511 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.342475 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zfv9\" (UniqueName: \"kubernetes.io/projected/cd039b73-b041-405b-9124-827ad244009a-kube-api-access-2zfv9\") pod \"maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7\" (UID: \"cd039b73-b041-405b-9124-827ad244009a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" Jul 17 22:09:14.342681 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.342546 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd039b73-b041-405b-9124-827ad244009a-maas-api-tls\") pod \"maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7\" (UID: \"cd039b73-b041-405b-9124-827ad244009a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" Jul 17 22:09:14.342681 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.342596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd039b73-b041-405b-9124-827ad244009a-openshift-service-ca\") pod \"maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7\" (UID: \"cd039b73-b041-405b-9124-827ad244009a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" Jul 17 22:09:14.342754 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:09:14.342681 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-serving-cert: secret "maas-api-e2e-mt-8962d973-serving-cert" not found Jul 17 22:09:14.342790 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:09:14.342761 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd039b73-b041-405b-9124-827ad244009a-maas-api-tls podName:cd039b73-b041-405b-9124-827ad244009a nodeName:}" failed. No retries permitted until 2026-07-17 22:09:14.842737816 +0000 UTC m=+2496.264457053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd039b73-b041-405b-9124-827ad244009a-maas-api-tls") pod "maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" (UID: "cd039b73-b041-405b-9124-827ad244009a") : secret "maas-api-e2e-mt-8962d973-serving-cert" not found Jul 17 22:09:14.343157 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.343141 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd039b73-b041-405b-9124-827ad244009a-openshift-service-ca\") pod \"maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7\" (UID: \"cd039b73-b041-405b-9124-827ad244009a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" Jul 17 22:09:14.353549 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.353529 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zfv9\" (UniqueName: \"kubernetes.io/projected/cd039b73-b041-405b-9124-827ad244009a-kube-api-access-2zfv9\") pod \"maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7\" (UID: \"cd039b73-b041-405b-9124-827ad244009a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" Jul 17 22:09:14.700954 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.700918 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:14.800345 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.800307 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:14.848723 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.848690 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd039b73-b041-405b-9124-827ad244009a-maas-api-tls\") pod \"maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7\" (UID: \"cd039b73-b041-405b-9124-827ad244009a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" Jul 17 22:09:14.851331 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.851307 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd039b73-b041-405b-9124-827ad244009a-maas-api-tls\") pod \"maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7\" (UID: \"cd039b73-b041-405b-9124-827ad244009a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" Jul 17 22:09:14.900602 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:14.900571 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:15.123899 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:15.123866 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" Jul 17 22:09:15.271209 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:15.271179 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7"] Jul 17 22:09:15.273217 ip-10-0-130-56 kubenswrapper[2580]: W0717 22:09:15.273189 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd039b73_b041_405b_9124_827ad244009a.slice/crio-aba1565d388e78954fa30c28c355138675b58efe4eed808359adf5bc4f0522c5 WatchSource:0}: Error finding container aba1565d388e78954fa30c28c355138675b58efe4eed808359adf5bc4f0522c5: Status 404 returned error can't find the container with id aba1565d388e78954fa30c28c355138675b58efe4eed808359adf5bc4f0522c5 Jul 17 22:09:15.274557 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:15.274540 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 22:09:16.226308 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:16.226211 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" event={"ID":"cd039b73-b041-405b-9124-827ad244009a","Type":"ContainerStarted","Data":"3388448720e20d3445f499e98854083acdbb967291216e0f7ea084a9161ae3b7"} Jul 17 22:09:16.226308 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:16.226275 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" event={"ID":"cd039b73-b041-405b-9124-827ad244009a","Type":"ContainerStarted","Data":"aba1565d388e78954fa30c28c355138675b58efe4eed808359adf5bc4f0522c5"} Jul 17 22:09:16.226308 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:16.226308 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" Jul 17 22:09:16.251175 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:16.251124 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" podStartSLOduration=1.743787632 podStartE2EDuration="2.251107899s" totalImagesPullingTime="507.320267ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:09:16.248932838 +0000 UTC m=+2497.670652105" watchObservedRunningTime="2026-07-17 22:09:16.251107899 +0000 UTC m=+2497.672827132" Jul 17 22:09:18.602019 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:18.601952 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:19.273645 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:19.273612 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:19.374086 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:19.374043 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:19.738548 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:19.738446 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:22.237955 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:22.237922 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" Jul 17 22:09:25.364638 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:25.364600 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:25.466138 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:25.466105 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:36.108308 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:36.108275 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7"] Jul 17 22:09:36.108936 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:36.108537 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" podUID="cd039b73-b041-405b-9124-827ad244009a" containerName="maas-api" containerID="cri-o://3388448720e20d3445f499e98854083acdbb967291216e0f7ea084a9161ae3b7" gracePeriod=30 Jul 17 22:09:36.139313 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:36.139265 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:36.334501 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:36.334473 2580 generic.go:356] "Generic (PLEG): container finished" podID="cd039b73-b041-405b-9124-827ad244009a" containerID="3388448720e20d3445f499e98854083acdbb967291216e0f7ea084a9161ae3b7" exitCode=0 Jul 17 22:09:36.334620 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:36.334520 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" event={"ID":"cd039b73-b041-405b-9124-827ad244009a","Type":"ContainerDied","Data":"3388448720e20d3445f499e98854083acdbb967291216e0f7ea084a9161ae3b7"} Jul 17 22:09:36.354266 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:36.354243 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" Jul 17 22:09:36.450638 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:36.450530 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zfv9\" (UniqueName: \"kubernetes.io/projected/cd039b73-b041-405b-9124-827ad244009a-kube-api-access-2zfv9\") pod \"cd039b73-b041-405b-9124-827ad244009a\" (UID: \"cd039b73-b041-405b-9124-827ad244009a\") " Jul 17 22:09:36.450638 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:36.450638 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd039b73-b041-405b-9124-827ad244009a-maas-api-tls\") pod \"cd039b73-b041-405b-9124-827ad244009a\" (UID: \"cd039b73-b041-405b-9124-827ad244009a\") " Jul 17 22:09:36.450894 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:36.450696 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd039b73-b041-405b-9124-827ad244009a-openshift-service-ca\") pod \"cd039b73-b041-405b-9124-827ad244009a\" (UID: \"cd039b73-b041-405b-9124-827ad244009a\") " Jul 17 22:09:36.451075 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:36.451043 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd039b73-b041-405b-9124-827ad244009a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd039b73-b041-405b-9124-827ad244009a" (UID: "cd039b73-b041-405b-9124-827ad244009a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:09:36.452845 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:36.452819 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd039b73-b041-405b-9124-827ad244009a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd039b73-b041-405b-9124-827ad244009a" (UID: "cd039b73-b041-405b-9124-827ad244009a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:09:36.452926 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:36.452859 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd039b73-b041-405b-9124-827ad244009a-kube-api-access-2zfv9" (OuterVolumeSpecName: "kube-api-access-2zfv9") pod "cd039b73-b041-405b-9124-827ad244009a" (UID: "cd039b73-b041-405b-9124-827ad244009a"). InnerVolumeSpecName "kube-api-access-2zfv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:09:36.551559 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:36.551523 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd039b73-b041-405b-9124-827ad244009a-maas-api-tls\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:09:36.551559 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:36.551552 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd039b73-b041-405b-9124-827ad244009a-openshift-service-ca\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:09:36.551559 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:36.551562 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zfv9\" (UniqueName: \"kubernetes.io/projected/cd039b73-b041-405b-9124-827ad244009a-kube-api-access-2zfv9\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:09:37.340915 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:37.340824 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" Jul 17 22:09:37.340915 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:37.340856 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7" event={"ID":"cd039b73-b041-405b-9124-827ad244009a","Type":"ContainerDied","Data":"aba1565d388e78954fa30c28c355138675b58efe4eed808359adf5bc4f0522c5"} Jul 17 22:09:37.341416 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:37.340916 2580 scope.go:117] "RemoveContainer" containerID="3388448720e20d3445f499e98854083acdbb967291216e0f7ea084a9161ae3b7" Jul 17 22:09:37.358651 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:37.358623 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7"] Jul 17 22:09:37.362537 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:37.362502 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8962d973-78dc4b5f56-pkhg7"] Jul 17 22:09:39.025599 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:39.025568 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd039b73-b041-405b-9124-827ad244009a" path="/var/lib/kubelet/pods/cd039b73-b041-405b-9124-827ad244009a/volumes" Jul 17 22:09:40.819427 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:40.819372 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:46.403154 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:46.403115 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:46.503912 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:46.503877 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:51.498599 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:51.498564 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:51.768902 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:51.768871 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:51.875383 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:51.875348 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:51.969906 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:51.969877 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:52.079252 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:52.079170 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:57.742849 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:57.742811 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:57.841549 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:57.841513 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:57.940161 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:57.940128 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:09:58.075270 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:09:58.075234 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:10:03.728525 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:03.728486 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:10:03.828825 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:03.828771 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:10:10.547783 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:10.547751 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:10:10.888576 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:10.888496 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:10:11.307491 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:11.307452 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:10:13.396009 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.395943 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:10:13.468869 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.468835 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:10:13.473099 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.473064 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s"] Jul 17 22:10:13.473464 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.473452 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerName="maas-api" Jul 17 22:10:13.473518 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.473466 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerName="maas-api" Jul 17 22:10:13.473518 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.473474 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerName="maas-api" Jul 17 22:10:13.473518 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.473479 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fbf3d87-47ce-4a7b-b944-6f02a23d31dd" containerName="maas-api" Jul 17 22:10:13.473518 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.473500 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd039b73-b041-405b-9124-827ad244009a" containerName="maas-api" Jul 17 22:10:13.473518 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.473505 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd039b73-b041-405b-9124-827ad244009a" containerName="maas-api" Jul 17 22:10:13.473686 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.473567 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd039b73-b041-405b-9124-827ad244009a" containerName="maas-api" Jul 17 22:10:13.476919 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.476896 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" Jul 17 22:10:13.479641 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.479615 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-539e6669-serving-cert\"" Jul 17 22:10:13.489684 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.489655 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s"] Jul 17 22:10:13.534093 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.534051 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:10:13.587775 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.587740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69bc6e0f-4306-4cab-8144-32f83560909b-maas-api-tls\") pod \"maas-api-e2e-mt-539e6669-55dd99977-w5x4s\" (UID: \"69bc6e0f-4306-4cab-8144-32f83560909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" Jul 17 22:10:13.587984 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.587796 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69bc6e0f-4306-4cab-8144-32f83560909b-openshift-service-ca\") pod \"maas-api-e2e-mt-539e6669-55dd99977-w5x4s\" (UID: \"69bc6e0f-4306-4cab-8144-32f83560909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" Jul 17 22:10:13.587984 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.587863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4crmg\" (UniqueName: \"kubernetes.io/projected/69bc6e0f-4306-4cab-8144-32f83560909b-kube-api-access-4crmg\") pod \"maas-api-e2e-mt-539e6669-55dd99977-w5x4s\" (UID: \"69bc6e0f-4306-4cab-8144-32f83560909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" Jul 17 22:10:13.688817 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.688731 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69bc6e0f-4306-4cab-8144-32f83560909b-maas-api-tls\") pod \"maas-api-e2e-mt-539e6669-55dd99977-w5x4s\" (UID: \"69bc6e0f-4306-4cab-8144-32f83560909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" Jul 17 22:10:13.688817 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.688793 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69bc6e0f-4306-4cab-8144-32f83560909b-openshift-service-ca\") pod \"maas-api-e2e-mt-539e6669-55dd99977-w5x4s\" (UID: \"69bc6e0f-4306-4cab-8144-32f83560909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" Jul 17 22:10:13.689009 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.688827 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4crmg\" (UniqueName: \"kubernetes.io/projected/69bc6e0f-4306-4cab-8144-32f83560909b-kube-api-access-4crmg\") pod \"maas-api-e2e-mt-539e6669-55dd99977-w5x4s\" (UID: \"69bc6e0f-4306-4cab-8144-32f83560909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" Jul 17 22:10:13.689606 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.689587 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69bc6e0f-4306-4cab-8144-32f83560909b-openshift-service-ca\") pod \"maas-api-e2e-mt-539e6669-55dd99977-w5x4s\" (UID: \"69bc6e0f-4306-4cab-8144-32f83560909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" Jul 17 22:10:13.691489 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.691464 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69bc6e0f-4306-4cab-8144-32f83560909b-maas-api-tls\") pod \"maas-api-e2e-mt-539e6669-55dd99977-w5x4s\" (UID: \"69bc6e0f-4306-4cab-8144-32f83560909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" Jul 17 22:10:13.697067 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.697043 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4crmg\" (UniqueName: \"kubernetes.io/projected/69bc6e0f-4306-4cab-8144-32f83560909b-kube-api-access-4crmg\") pod \"maas-api-e2e-mt-539e6669-55dd99977-w5x4s\" (UID: \"69bc6e0f-4306-4cab-8144-32f83560909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" Jul 17 22:10:13.722994 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.722961 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:10:13.790225 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.790190 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" Jul 17 22:10:13.927004 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:13.926980 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s"] Jul 17 22:10:13.928984 ip-10-0-130-56 kubenswrapper[2580]: W0717 22:10:13.928959 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69bc6e0f_4306_4cab_8144_32f83560909b.slice/crio-f7e92a7a0e08521ac27168d178010a44c5b58163aa1637e1dcedd41f7aab4af1 WatchSource:0}: Error finding container f7e92a7a0e08521ac27168d178010a44c5b58163aa1637e1dcedd41f7aab4af1: Status 404 returned error can't find the container with id f7e92a7a0e08521ac27168d178010a44c5b58163aa1637e1dcedd41f7aab4af1 Jul 17 22:10:14.145158 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:14.145129 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:10:14.249919 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:14.249878 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:10:14.543033 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:14.542998 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" event={"ID":"69bc6e0f-4306-4cab-8144-32f83560909b","Type":"ContainerStarted","Data":"04e9ac5b2aba083c0db2bdf6a023be55082318316e2b9784813b0e6fcc099d51"} Jul 17 22:10:14.543492 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:14.543046 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" event={"ID":"69bc6e0f-4306-4cab-8144-32f83560909b","Type":"ContainerStarted","Data":"f7e92a7a0e08521ac27168d178010a44c5b58163aa1637e1dcedd41f7aab4af1"} Jul 17 22:10:14.543492 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:14.543143 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" Jul 17 22:10:14.561956 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:14.561896 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" podStartSLOduration=1.178688919 podStartE2EDuration="1.561880532s" totalImagesPullingTime="383.191613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:10:14.559548641 +0000 UTC m=+2555.981267897" watchObservedRunningTime="2026-07-17 22:10:14.561880532 +0000 UTC m=+2555.983599765" Jul 17 22:10:17.888955 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:17.888902 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:10:20.560860 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:20.560826 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" Jul 17 22:10:54.758201 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:10:54.758154 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:11:12.471747 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:12.471696 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:11:32.222747 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.222715 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s"] Jul 17 22:11:32.223322 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.222961 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" podUID="69bc6e0f-4306-4cab-8144-32f83560909b" containerName="maas-api" containerID="cri-o://04e9ac5b2aba083c0db2bdf6a023be55082318316e2b9784813b0e6fcc099d51" gracePeriod=30 Jul 17 22:11:32.247083 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.247058 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:11:32.473161 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.473089 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" Jul 17 22:11:32.528660 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.528627 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69bc6e0f-4306-4cab-8144-32f83560909b-maas-api-tls\") pod \"69bc6e0f-4306-4cab-8144-32f83560909b\" (UID: \"69bc6e0f-4306-4cab-8144-32f83560909b\") " Jul 17 22:11:32.528860 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.528673 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69bc6e0f-4306-4cab-8144-32f83560909b-openshift-service-ca\") pod \"69bc6e0f-4306-4cab-8144-32f83560909b\" (UID: \"69bc6e0f-4306-4cab-8144-32f83560909b\") " Jul 17 22:11:32.528860 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.528705 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4crmg\" (UniqueName: \"kubernetes.io/projected/69bc6e0f-4306-4cab-8144-32f83560909b-kube-api-access-4crmg\") pod \"69bc6e0f-4306-4cab-8144-32f83560909b\" (UID: \"69bc6e0f-4306-4cab-8144-32f83560909b\") " Jul 17 22:11:32.529083 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.529058 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69bc6e0f-4306-4cab-8144-32f83560909b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "69bc6e0f-4306-4cab-8144-32f83560909b" (UID: "69bc6e0f-4306-4cab-8144-32f83560909b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:11:32.530954 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.530931 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69bc6e0f-4306-4cab-8144-32f83560909b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "69bc6e0f-4306-4cab-8144-32f83560909b" (UID: "69bc6e0f-4306-4cab-8144-32f83560909b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:11:32.531048 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.531029 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69bc6e0f-4306-4cab-8144-32f83560909b-kube-api-access-4crmg" (OuterVolumeSpecName: "kube-api-access-4crmg") pod "69bc6e0f-4306-4cab-8144-32f83560909b" (UID: "69bc6e0f-4306-4cab-8144-32f83560909b"). InnerVolumeSpecName "kube-api-access-4crmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:11:32.629674 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.629636 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69bc6e0f-4306-4cab-8144-32f83560909b-maas-api-tls\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:11:32.629674 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.629668 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69bc6e0f-4306-4cab-8144-32f83560909b-openshift-service-ca\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:11:32.629674 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.629681 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4crmg\" (UniqueName: \"kubernetes.io/projected/69bc6e0f-4306-4cab-8144-32f83560909b-kube-api-access-4crmg\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:11:32.967513 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.967481 2580 generic.go:356] "Generic (PLEG): container finished" podID="69bc6e0f-4306-4cab-8144-32f83560909b" containerID="04e9ac5b2aba083c0db2bdf6a023be55082318316e2b9784813b0e6fcc099d51" exitCode=0 Jul 17 22:11:32.967730 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.967526 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" Jul 17 22:11:32.967730 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.967577 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" event={"ID":"69bc6e0f-4306-4cab-8144-32f83560909b","Type":"ContainerDied","Data":"04e9ac5b2aba083c0db2bdf6a023be55082318316e2b9784813b0e6fcc099d51"} Jul 17 22:11:32.967730 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.967639 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s" event={"ID":"69bc6e0f-4306-4cab-8144-32f83560909b","Type":"ContainerDied","Data":"f7e92a7a0e08521ac27168d178010a44c5b58163aa1637e1dcedd41f7aab4af1"} Jul 17 22:11:32.967730 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.967675 2580 scope.go:117] "RemoveContainer" containerID="04e9ac5b2aba083c0db2bdf6a023be55082318316e2b9784813b0e6fcc099d51" Jul 17 22:11:32.977746 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.977726 2580 scope.go:117] "RemoveContainer" containerID="04e9ac5b2aba083c0db2bdf6a023be55082318316e2b9784813b0e6fcc099d51" Jul 17 22:11:32.978037 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:11:32.978014 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e9ac5b2aba083c0db2bdf6a023be55082318316e2b9784813b0e6fcc099d51\": container with ID starting with 04e9ac5b2aba083c0db2bdf6a023be55082318316e2b9784813b0e6fcc099d51 not found: ID does not exist" containerID="04e9ac5b2aba083c0db2bdf6a023be55082318316e2b9784813b0e6fcc099d51" Jul 17 22:11:32.978107 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.978045 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e9ac5b2aba083c0db2bdf6a023be55082318316e2b9784813b0e6fcc099d51"} err="failed to get container status \"04e9ac5b2aba083c0db2bdf6a023be55082318316e2b9784813b0e6fcc099d51\": rpc error: code = NotFound desc = could not find container \"04e9ac5b2aba083c0db2bdf6a023be55082318316e2b9784813b0e6fcc099d51\": container with ID starting with 04e9ac5b2aba083c0db2bdf6a023be55082318316e2b9784813b0e6fcc099d51 not found: ID does not exist" Jul 17 22:11:32.989433 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.989382 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s"] Jul 17 22:11:32.993298 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:32.993278 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-539e6669-55dd99977-w5x4s"] Jul 17 22:11:33.023936 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:33.023901 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69bc6e0f-4306-4cab-8144-32f83560909b" path="/var/lib/kubelet/pods/69bc6e0f-4306-4cab-8144-32f83560909b/volumes" Jul 17 22:11:33.145787 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:33.145757 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:11:33.248110 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:33.248023 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:11:36.890332 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:36.890289 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:11:41.912291 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:11:41.912251 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:12:18.442683 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:18.442647 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:12:18.508726 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:18.508688 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:12:18.843424 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:18.843378 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:12:19.183662 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:19.183566 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:12:24.667138 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:24.667090 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:12:38.891745 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:38.891648 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:12:38.952155 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:38.952127 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:12:39.208670 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:39.208577 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 22:12:39.219222 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:39.219196 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 22:12:49.205030 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:49.204988 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:12:49.432508 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:49.432473 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:12:50.536522 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:50.536476 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:12:50.639140 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:50.639105 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:12:50.742045 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:50.742010 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:12:51.445563 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:51.445531 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:12:51.543811 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:51.543779 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:12:51.643776 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:51.643741 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:12:53.038359 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:53.038315 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:12:59.359782 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:59.359744 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:12:59.457170 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:12:59.457133 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:00.078845 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:00.078813 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:05.789181 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:05.789145 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:05.889855 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:05.889826 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:05.979999 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:05.979968 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:16.497284 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:16.497245 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:16.845873 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:16.845836 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:37.832437 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:37.832373 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:43.706823 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:43.706783 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:43.808595 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:43.808560 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:44.055987 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:44.055950 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:44.396050 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:44.395965 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:44.526980 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:44.526948 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:47.506840 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:47.506810 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:47.595688 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:47.595653 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:47.602672 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:47.602644 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf"] Jul 17 22:13:47.603178 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:47.603164 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69bc6e0f-4306-4cab-8144-32f83560909b" containerName="maas-api" Jul 17 22:13:47.603222 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:47.603183 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="69bc6e0f-4306-4cab-8144-32f83560909b" containerName="maas-api" Jul 17 22:13:47.603308 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:47.603296 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="69bc6e0f-4306-4cab-8144-32f83560909b" containerName="maas-api" Jul 17 22:13:47.606712 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:47.606692 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" Jul 17 22:13:47.609271 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:47.609251 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-be8c97-serving-cert\"" Jul 17 22:13:47.616813 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:47.616786 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf"] Jul 17 22:13:47.654892 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:47.654865 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p57w\" (UniqueName: \"kubernetes.io/projected/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-kube-api-access-2p57w\") pod \"maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf\" (UID: \"68e568d7-8c1a-49d4-bb7d-06fbbdd54460\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" Jul 17 22:13:47.655058 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:47.654964 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-maas-api-tls\") pod \"maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf\" (UID: \"68e568d7-8c1a-49d4-bb7d-06fbbdd54460\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" Jul 17 22:13:47.655058 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:47.655004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-openshift-service-ca\") pod \"maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf\" (UID: \"68e568d7-8c1a-49d4-bb7d-06fbbdd54460\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" Jul 17 22:13:47.669709 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:47.669676 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:47.755677 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:47.755645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p57w\" (UniqueName: \"kubernetes.io/projected/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-kube-api-access-2p57w\") pod \"maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf\" (UID: \"68e568d7-8c1a-49d4-bb7d-06fbbdd54460\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" Jul 17 22:13:47.755858 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:47.755704 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-maas-api-tls\") pod \"maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf\" (UID: \"68e568d7-8c1a-49d4-bb7d-06fbbdd54460\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" Jul 17 22:13:47.755858 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:47.755735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-openshift-service-ca\") pod \"maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf\" (UID: \"68e568d7-8c1a-49d4-bb7d-06fbbdd54460\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" Jul 17 22:13:47.755946 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:13:47.755869 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-serving-cert: secret "maas-api-e2e-inf-a-be8c97-serving-cert" not found Jul 17 22:13:47.755990 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:13:47.755948 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-maas-api-tls podName:68e568d7-8c1a-49d4-bb7d-06fbbdd54460 nodeName:}" failed. No retries permitted until 2026-07-17 22:13:48.255928578 +0000 UTC m=+2769.677647789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-maas-api-tls") pod "maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" (UID: "68e568d7-8c1a-49d4-bb7d-06fbbdd54460") : secret "maas-api-e2e-inf-a-be8c97-serving-cert" not found Jul 17 22:13:47.756491 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:47.756473 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-openshift-service-ca\") pod \"maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf\" (UID: \"68e568d7-8c1a-49d4-bb7d-06fbbdd54460\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" Jul 17 22:13:47.766431 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:47.766411 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p57w\" (UniqueName: \"kubernetes.io/projected/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-kube-api-access-2p57w\") pod \"maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf\" (UID: \"68e568d7-8c1a-49d4-bb7d-06fbbdd54460\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" Jul 17 22:13:48.260979 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:48.260886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-maas-api-tls\") pod \"maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf\" (UID: \"68e568d7-8c1a-49d4-bb7d-06fbbdd54460\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" Jul 17 22:13:48.261157 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:13:48.261046 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-serving-cert: secret "maas-api-e2e-inf-a-be8c97-serving-cert" not found Jul 17 22:13:48.261157 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:13:48.261117 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-maas-api-tls podName:68e568d7-8c1a-49d4-bb7d-06fbbdd54460 nodeName:}" failed. No retries permitted until 2026-07-17 22:13:49.261098452 +0000 UTC m=+2770.682817663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-maas-api-tls") pod "maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" (UID: "68e568d7-8c1a-49d4-bb7d-06fbbdd54460") : secret "maas-api-e2e-inf-a-be8c97-serving-cert" not found Jul 17 22:13:48.261729 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:48.261704 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:48.363280 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:48.363245 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:48.463777 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:48.463738 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:49.270208 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:49.270171 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-maas-api-tls\") pod \"maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf\" (UID: \"68e568d7-8c1a-49d4-bb7d-06fbbdd54460\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" Jul 17 22:13:49.272783 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:49.272750 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-maas-api-tls\") pod \"maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf\" (UID: \"68e568d7-8c1a-49d4-bb7d-06fbbdd54460\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" Jul 17 22:13:49.418639 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:49.418603 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" Jul 17 22:13:49.557891 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:49.557858 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf"] Jul 17 22:13:49.559840 ip-10-0-130-56 kubenswrapper[2580]: W0717 22:13:49.559808 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e568d7_8c1a_49d4_bb7d_06fbbdd54460.slice/crio-4eb7720a98e12cb65c14d312f27d1d838f3c8b23364c15be09269792bdc6515a WatchSource:0}: Error finding container 4eb7720a98e12cb65c14d312f27d1d838f3c8b23364c15be09269792bdc6515a: Status 404 returned error can't find the container with id 4eb7720a98e12cb65c14d312f27d1d838f3c8b23364c15be09269792bdc6515a Jul 17 22:13:49.705171 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:49.705134 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" event={"ID":"68e568d7-8c1a-49d4-bb7d-06fbbdd54460","Type":"ContainerStarted","Data":"4eb7720a98e12cb65c14d312f27d1d838f3c8b23364c15be09269792bdc6515a"} Jul 17 22:13:50.711652 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:50.711612 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" event={"ID":"68e568d7-8c1a-49d4-bb7d-06fbbdd54460","Type":"ContainerStarted","Data":"4a0bf9cca98f5fc952847bf3dcdd6bba0563f688641c6bdcc8c0a8771c3f8858"} Jul 17 22:13:50.712051 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:50.711736 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" Jul 17 22:13:50.733707 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:50.733655 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" podStartSLOduration=3.241834859 podStartE2EDuration="3.733640762s" totalImagesPullingTime="491.805903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:13:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:13:50.732784379 +0000 UTC m=+2772.154503622" watchObservedRunningTime="2026-07-17 22:13:50.733640762 +0000 UTC m=+2772.155359995" Jul 17 22:13:52.029651 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:52.029620 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:52.360204 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:52.360121 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:52.715357 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:52.715279 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:52.814192 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:52.814159 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:55.581746 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:55.581711 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:55.666182 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:55.666123 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:55.667061 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:55.667036 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg"] Jul 17 22:13:55.670636 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:55.670618 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" Jul 17 22:13:55.673103 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:55.673078 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b09be9-serving-cert\"" Jul 17 22:13:55.680864 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:55.680834 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg"] Jul 17 22:13:55.732864 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:55.732830 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:55.734054 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:55.734023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlmv6\" (UniqueName: \"kubernetes.io/projected/1cebc564-b122-40d4-8670-0663f0cf04be-kube-api-access-qlmv6\") pod \"maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg\" (UID: \"1cebc564-b122-40d4-8670-0663f0cf04be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" Jul 17 22:13:55.734208 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:55.734092 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cebc564-b122-40d4-8670-0663f0cf04be-maas-api-tls\") pod \"maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg\" (UID: \"1cebc564-b122-40d4-8670-0663f0cf04be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" Jul 17 22:13:55.734208 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:55.734195 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cebc564-b122-40d4-8670-0663f0cf04be-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg\" (UID: \"1cebc564-b122-40d4-8670-0663f0cf04be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" Jul 17 22:13:55.835019 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:55.834919 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlmv6\" (UniqueName: \"kubernetes.io/projected/1cebc564-b122-40d4-8670-0663f0cf04be-kube-api-access-qlmv6\") pod \"maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg\" (UID: \"1cebc564-b122-40d4-8670-0663f0cf04be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" Jul 17 22:13:55.835019 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:55.834969 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cebc564-b122-40d4-8670-0663f0cf04be-maas-api-tls\") pod \"maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg\" (UID: \"1cebc564-b122-40d4-8670-0663f0cf04be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" Jul 17 22:13:55.835019 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:55.835017 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cebc564-b122-40d4-8670-0663f0cf04be-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg\" (UID: \"1cebc564-b122-40d4-8670-0663f0cf04be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" Jul 17 22:13:55.835296 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:13:55.835144 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-serving-cert: secret "maas-api-e2e-inf-b-b09be9-serving-cert" not found Jul 17 22:13:55.835296 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:13:55.835225 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cebc564-b122-40d4-8670-0663f0cf04be-maas-api-tls podName:1cebc564-b122-40d4-8670-0663f0cf04be nodeName:}" failed. No retries permitted until 2026-07-17 22:13:56.335207809 +0000 UTC m=+2777.756927024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1cebc564-b122-40d4-8670-0663f0cf04be-maas-api-tls") pod "maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" (UID: "1cebc564-b122-40d4-8670-0663f0cf04be") : secret "maas-api-e2e-inf-b-b09be9-serving-cert" not found Jul 17 22:13:55.835858 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:55.835833 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cebc564-b122-40d4-8670-0663f0cf04be-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg\" (UID: \"1cebc564-b122-40d4-8670-0663f0cf04be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" Jul 17 22:13:55.849006 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:55.848980 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlmv6\" (UniqueName: \"kubernetes.io/projected/1cebc564-b122-40d4-8670-0663f0cf04be-kube-api-access-qlmv6\") pod \"maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg\" (UID: \"1cebc564-b122-40d4-8670-0663f0cf04be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" Jul 17 22:13:56.018815 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:56.018773 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:56.230626 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:56.230507 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:56.331583 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:56.331555 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:13:56.339939 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:56.339915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cebc564-b122-40d4-8670-0663f0cf04be-maas-api-tls\") pod \"maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg\" (UID: \"1cebc564-b122-40d4-8670-0663f0cf04be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" Jul 17 22:13:56.342597 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:56.342575 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cebc564-b122-40d4-8670-0663f0cf04be-maas-api-tls\") pod \"maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg\" (UID: \"1cebc564-b122-40d4-8670-0663f0cf04be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" Jul 17 22:13:56.583913 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:56.583876 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" Jul 17 22:13:56.724796 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:56.724636 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" Jul 17 22:13:56.724990 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:56.724847 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg"] Jul 17 22:13:56.727987 ip-10-0-130-56 kubenswrapper[2580]: W0717 22:13:56.727959 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cebc564_b122_40d4_8670_0663f0cf04be.slice/crio-2ef9116ed02540c5a1c0531e787d6734035da2a0a8854f16312c582a5ef76a73 WatchSource:0}: Error finding container 2ef9116ed02540c5a1c0531e787d6734035da2a0a8854f16312c582a5ef76a73: Status 404 returned error can't find the container with id 2ef9116ed02540c5a1c0531e787d6734035da2a0a8854f16312c582a5ef76a73 Jul 17 22:13:56.750570 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:56.750518 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" event={"ID":"1cebc564-b122-40d4-8670-0663f0cf04be","Type":"ContainerStarted","Data":"2ef9116ed02540c5a1c0531e787d6734035da2a0a8854f16312c582a5ef76a73"} Jul 17 22:13:57.756985 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:57.756948 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" event={"ID":"1cebc564-b122-40d4-8670-0663f0cf04be","Type":"ContainerStarted","Data":"4de9f51ae4427565321f99edac74035d112c4f8448702925506c86205628dd3b"} Jul 17 22:13:57.757385 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:57.757059 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" Jul 17 22:13:57.773261 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:13:57.773188 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" podStartSLOduration=2.427258814 podStartE2EDuration="2.773172393s" totalImagesPullingTime="345.913579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:13:57.772634018 +0000 UTC m=+2779.194353249" watchObservedRunningTime="2026-07-17 22:13:57.773172393 +0000 UTC m=+2779.194891625" Jul 17 22:14:00.084112 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:14:00.084003 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:14:03.767931 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:14:03.767902 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" Jul 17 22:14:09.145218 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:14:09.145182 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:14:09.246121 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:14:09.246067 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:14:09.926228 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:14:09.926197 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:14:17.111646 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:14:17.111606 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:14:18.290741 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:14:18.290709 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:14:27.514664 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:14:27.514632 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:14:53.434546 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:14:53.434508 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:14:54.440749 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:14:54.440708 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:14:55.080303 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:14:55.080258 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:14:55.185857 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:14:55.185823 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:05.060068 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:05.060032 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf"] Jul 17 22:15:05.060571 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:05.060274 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" podUID="68e568d7-8c1a-49d4-bb7d-06fbbdd54460" containerName="maas-api" containerID="cri-o://4a0bf9cca98f5fc952847bf3dcdd6bba0563f688641c6bdcc8c0a8771c3f8858" gracePeriod=30 Jul 17 22:15:05.090812 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:05.090731 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:05.313606 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:05.313535 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" Jul 17 22:15:05.393353 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:05.393316 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-maas-api-tls\") pod \"68e568d7-8c1a-49d4-bb7d-06fbbdd54460\" (UID: \"68e568d7-8c1a-49d4-bb7d-06fbbdd54460\") " Jul 17 22:15:05.393557 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:05.393372 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-openshift-service-ca\") pod \"68e568d7-8c1a-49d4-bb7d-06fbbdd54460\" (UID: \"68e568d7-8c1a-49d4-bb7d-06fbbdd54460\") " Jul 17 22:15:05.393557 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:05.393519 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p57w\" (UniqueName: \"kubernetes.io/projected/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-kube-api-access-2p57w\") pod \"68e568d7-8c1a-49d4-bb7d-06fbbdd54460\" (UID: \"68e568d7-8c1a-49d4-bb7d-06fbbdd54460\") " Jul 17 22:15:05.393739 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:05.393717 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68e568d7-8c1a-49d4-bb7d-06fbbdd54460" (UID: "68e568d7-8c1a-49d4-bb7d-06fbbdd54460"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:15:05.393816 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:05.393804 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-openshift-service-ca\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:15:05.395588 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:05.395563 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68e568d7-8c1a-49d4-bb7d-06fbbdd54460" (UID: "68e568d7-8c1a-49d4-bb7d-06fbbdd54460"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:15:05.395740 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:05.395717 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-kube-api-access-2p57w" (OuterVolumeSpecName: "kube-api-access-2p57w") pod "68e568d7-8c1a-49d4-bb7d-06fbbdd54460" (UID: "68e568d7-8c1a-49d4-bb7d-06fbbdd54460"). InnerVolumeSpecName "kube-api-access-2p57w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:15:05.494399 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:05.494364 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-maas-api-tls\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:15:05.494568 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:05.494420 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p57w\" (UniqueName: \"kubernetes.io/projected/68e568d7-8c1a-49d4-bb7d-06fbbdd54460-kube-api-access-2p57w\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:15:06.122726 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:06.122692 2580 generic.go:356] "Generic (PLEG): container finished" podID="68e568d7-8c1a-49d4-bb7d-06fbbdd54460" containerID="4a0bf9cca98f5fc952847bf3dcdd6bba0563f688641c6bdcc8c0a8771c3f8858" exitCode=0 Jul 17 22:15:06.123098 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:06.122740 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" Jul 17 22:15:06.123098 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:06.122779 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" event={"ID":"68e568d7-8c1a-49d4-bb7d-06fbbdd54460","Type":"ContainerDied","Data":"4a0bf9cca98f5fc952847bf3dcdd6bba0563f688641c6bdcc8c0a8771c3f8858"} Jul 17 22:15:06.123098 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:06.122830 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf" event={"ID":"68e568d7-8c1a-49d4-bb7d-06fbbdd54460","Type":"ContainerDied","Data":"4eb7720a98e12cb65c14d312f27d1d838f3c8b23364c15be09269792bdc6515a"} Jul 17 22:15:06.123098 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:06.122851 2580 scope.go:117] "RemoveContainer" containerID="4a0bf9cca98f5fc952847bf3dcdd6bba0563f688641c6bdcc8c0a8771c3f8858" Jul 17 22:15:06.132700 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:06.132684 2580 scope.go:117] "RemoveContainer" containerID="4a0bf9cca98f5fc952847bf3dcdd6bba0563f688641c6bdcc8c0a8771c3f8858" Jul 17 22:15:06.132981 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:15:06.132965 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a0bf9cca98f5fc952847bf3dcdd6bba0563f688641c6bdcc8c0a8771c3f8858\": container with ID starting with 4a0bf9cca98f5fc952847bf3dcdd6bba0563f688641c6bdcc8c0a8771c3f8858 not found: ID does not exist" containerID="4a0bf9cca98f5fc952847bf3dcdd6bba0563f688641c6bdcc8c0a8771c3f8858" Jul 17 22:15:06.133019 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:06.132990 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a0bf9cca98f5fc952847bf3dcdd6bba0563f688641c6bdcc8c0a8771c3f8858"} err="failed to get container status \"4a0bf9cca98f5fc952847bf3dcdd6bba0563f688641c6bdcc8c0a8771c3f8858\": rpc error: code = NotFound desc = could not find container \"4a0bf9cca98f5fc952847bf3dcdd6bba0563f688641c6bdcc8c0a8771c3f8858\": container with ID starting with 4a0bf9cca98f5fc952847bf3dcdd6bba0563f688641c6bdcc8c0a8771c3f8858 not found: ID does not exist" Jul 17 22:15:06.145458 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:06.145433 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf"] Jul 17 22:15:06.151197 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:06.151171 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-be8c97-86d85c87d-qqbzf"] Jul 17 22:15:07.023963 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:07.023919 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68e568d7-8c1a-49d4-bb7d-06fbbdd54460" path="/var/lib/kubelet/pods/68e568d7-8c1a-49d4-bb7d-06fbbdd54460/volumes" Jul 17 22:15:09.643784 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:09.643750 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:15.276580 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:15.276548 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:15.377190 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:15.377157 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:20.332775 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:20.332714 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:20.571158 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:20.571119 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:21.190657 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:21.190622 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:21.291302 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:21.291267 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:31.065342 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:31.065228 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg"] Jul 17 22:15:31.065924 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:31.065542 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" podUID="1cebc564-b122-40d4-8670-0663f0cf04be" containerName="maas-api" containerID="cri-o://4de9f51ae4427565321f99edac74035d112c4f8448702925506c86205628dd3b" gracePeriod=30 Jul 17 22:15:31.092859 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:31.092826 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:31.267229 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:31.267193 2580 generic.go:356] "Generic (PLEG): container finished" podID="1cebc564-b122-40d4-8670-0663f0cf04be" containerID="4de9f51ae4427565321f99edac74035d112c4f8448702925506c86205628dd3b" exitCode=0 Jul 17 22:15:31.267460 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:31.267265 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" event={"ID":"1cebc564-b122-40d4-8670-0663f0cf04be","Type":"ContainerDied","Data":"4de9f51ae4427565321f99edac74035d112c4f8448702925506c86205628dd3b"} Jul 17 22:15:31.321292 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:31.321225 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" Jul 17 22:15:31.348194 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:31.348162 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cebc564-b122-40d4-8670-0663f0cf04be-maas-api-tls\") pod \"1cebc564-b122-40d4-8670-0663f0cf04be\" (UID: \"1cebc564-b122-40d4-8670-0663f0cf04be\") " Jul 17 22:15:31.348371 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:31.348204 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cebc564-b122-40d4-8670-0663f0cf04be-openshift-service-ca\") pod \"1cebc564-b122-40d4-8670-0663f0cf04be\" (UID: \"1cebc564-b122-40d4-8670-0663f0cf04be\") " Jul 17 22:15:31.348371 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:31.348332 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlmv6\" (UniqueName: \"kubernetes.io/projected/1cebc564-b122-40d4-8670-0663f0cf04be-kube-api-access-qlmv6\") pod \"1cebc564-b122-40d4-8670-0663f0cf04be\" (UID: \"1cebc564-b122-40d4-8670-0663f0cf04be\") " Jul 17 22:15:31.348684 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:31.348652 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cebc564-b122-40d4-8670-0663f0cf04be-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1cebc564-b122-40d4-8670-0663f0cf04be" (UID: "1cebc564-b122-40d4-8670-0663f0cf04be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:15:31.350618 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:31.350589 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cebc564-b122-40d4-8670-0663f0cf04be-kube-api-access-qlmv6" (OuterVolumeSpecName: "kube-api-access-qlmv6") pod "1cebc564-b122-40d4-8670-0663f0cf04be" (UID: "1cebc564-b122-40d4-8670-0663f0cf04be"). InnerVolumeSpecName "kube-api-access-qlmv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:15:31.350717 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:31.350666 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cebc564-b122-40d4-8670-0663f0cf04be-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1cebc564-b122-40d4-8670-0663f0cf04be" (UID: "1cebc564-b122-40d4-8670-0663f0cf04be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:15:31.450054 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:31.450003 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlmv6\" (UniqueName: \"kubernetes.io/projected/1cebc564-b122-40d4-8670-0663f0cf04be-kube-api-access-qlmv6\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:15:31.450054 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:31.450049 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cebc564-b122-40d4-8670-0663f0cf04be-maas-api-tls\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:15:31.450054 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:31.450059 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cebc564-b122-40d4-8670-0663f0cf04be-openshift-service-ca\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:15:32.273218 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:32.273189 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" Jul 17 22:15:32.273680 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:32.273230 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg" event={"ID":"1cebc564-b122-40d4-8670-0663f0cf04be","Type":"ContainerDied","Data":"2ef9116ed02540c5a1c0531e787d6734035da2a0a8854f16312c582a5ef76a73"} Jul 17 22:15:32.273680 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:32.273294 2580 scope.go:117] "RemoveContainer" containerID="4de9f51ae4427565321f99edac74035d112c4f8448702925506c86205628dd3b" Jul 17 22:15:32.294321 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:32.294290 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg"] Jul 17 22:15:32.298673 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:32.298639 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b09be9-f88d56878-f4vvg"] Jul 17 22:15:33.023625 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:33.023591 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cebc564-b122-40d4-8670-0663f0cf04be" path="/var/lib/kubelet/pods/1cebc564-b122-40d4-8670-0663f0cf04be/volumes" Jul 17 22:15:35.777796 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:35.777766 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:41.562838 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:41.562805 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:41.662701 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:41.662666 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:46.480218 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:46.480184 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:46.711125 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:46.711091 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:47.066365 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:47.066317 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:47.382960 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:47.382872 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:47.566917 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:47.566881 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:50.225641 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.225606 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:50.304221 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.304192 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:50.309595 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.309569 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw"] Jul 17 22:15:50.309924 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.309911 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68e568d7-8c1a-49d4-bb7d-06fbbdd54460" containerName="maas-api" Jul 17 22:15:50.309972 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.309925 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="68e568d7-8c1a-49d4-bb7d-06fbbdd54460" containerName="maas-api" Jul 17 22:15:50.309972 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.309950 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cebc564-b122-40d4-8670-0663f0cf04be" containerName="maas-api" Jul 17 22:15:50.309972 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.309955 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cebc564-b122-40d4-8670-0663f0cf04be" containerName="maas-api" Jul 17 22:15:50.310084 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.310034 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cebc564-b122-40d4-8670-0663f0cf04be" containerName="maas-api" Jul 17 22:15:50.310084 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.310045 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="68e568d7-8c1a-49d4-bb7d-06fbbdd54460" containerName="maas-api" Jul 17 22:15:50.312975 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.312960 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" Jul 17 22:15:50.315378 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.315356 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-95299c-serving-cert\"" Jul 17 22:15:50.323598 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.323574 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw"] Jul 17 22:15:50.411107 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.411074 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:50.421329 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.421296 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357d1923-afdc-4e83-893a-374e855a14e2-maas-api-tls\") pod \"maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw\" (UID: \"357d1923-afdc-4e83-893a-374e855a14e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" Jul 17 22:15:50.421511 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.421332 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/357d1923-afdc-4e83-893a-374e855a14e2-openshift-service-ca\") pod \"maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw\" (UID: \"357d1923-afdc-4e83-893a-374e855a14e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" Jul 17 22:15:50.421511 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.421373 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vh8d\" (UniqueName: \"kubernetes.io/projected/357d1923-afdc-4e83-893a-374e855a14e2-kube-api-access-8vh8d\") pod \"maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw\" (UID: \"357d1923-afdc-4e83-893a-374e855a14e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" Jul 17 22:15:50.522778 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.522743 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357d1923-afdc-4e83-893a-374e855a14e2-maas-api-tls\") pod \"maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw\" (UID: \"357d1923-afdc-4e83-893a-374e855a14e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" Jul 17 22:15:50.522962 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.522782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/357d1923-afdc-4e83-893a-374e855a14e2-openshift-service-ca\") pod \"maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw\" (UID: \"357d1923-afdc-4e83-893a-374e855a14e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" Jul 17 22:15:50.522962 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:15:50.522902 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-serving-cert: secret "maas-api-e2e-api-a-95299c-serving-cert" not found Jul 17 22:15:50.523048 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:15:50.522970 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/357d1923-afdc-4e83-893a-374e855a14e2-maas-api-tls podName:357d1923-afdc-4e83-893a-374e855a14e2 nodeName:}" failed. No retries permitted until 2026-07-17 22:15:51.022951348 +0000 UTC m=+2892.444670559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/357d1923-afdc-4e83-893a-374e855a14e2-maas-api-tls") pod "maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" (UID: "357d1923-afdc-4e83-893a-374e855a14e2") : secret "maas-api-e2e-api-a-95299c-serving-cert" not found Jul 17 22:15:50.523048 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.522908 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vh8d\" (UniqueName: \"kubernetes.io/projected/357d1923-afdc-4e83-893a-374e855a14e2-kube-api-access-8vh8d\") pod \"maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw\" (UID: \"357d1923-afdc-4e83-893a-374e855a14e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" Jul 17 22:15:50.523461 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.523445 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/357d1923-afdc-4e83-893a-374e855a14e2-openshift-service-ca\") pod \"maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw\" (UID: \"357d1923-afdc-4e83-893a-374e855a14e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" Jul 17 22:15:50.532225 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.532196 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vh8d\" (UniqueName: \"kubernetes.io/projected/357d1923-afdc-4e83-893a-374e855a14e2-kube-api-access-8vh8d\") pod \"maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw\" (UID: \"357d1923-afdc-4e83-893a-374e855a14e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" Jul 17 22:15:50.705647 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:50.705610 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:51.028806 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:51.028775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357d1923-afdc-4e83-893a-374e855a14e2-maas-api-tls\") pod \"maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw\" (UID: \"357d1923-afdc-4e83-893a-374e855a14e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" Jul 17 22:15:51.031568 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:51.031541 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357d1923-afdc-4e83-893a-374e855a14e2-maas-api-tls\") pod \"maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw\" (UID: \"357d1923-afdc-4e83-893a-374e855a14e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" Jul 17 22:15:51.047981 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:51.047955 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:51.149747 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:51.149712 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:51.224290 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:51.224251 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" Jul 17 22:15:51.367598 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:51.367575 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw"] Jul 17 22:15:51.369544 ip-10-0-130-56 kubenswrapper[2580]: W0717 22:15:51.369500 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357d1923_afdc_4e83_893a_374e855a14e2.slice/crio-0fed69c60c809e2b4fb540e6adf5ca5f458943342ab64ce0ef8d9bd8bc0c2b8b WatchSource:0}: Error finding container 0fed69c60c809e2b4fb540e6adf5ca5f458943342ab64ce0ef8d9bd8bc0c2b8b: Status 404 returned error can't find the container with id 0fed69c60c809e2b4fb540e6adf5ca5f458943342ab64ce0ef8d9bd8bc0c2b8b Jul 17 22:15:51.370926 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:51.370904 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 22:15:52.382272 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:52.382232 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" event={"ID":"357d1923-afdc-4e83-893a-374e855a14e2","Type":"ContainerStarted","Data":"31fb3d9f6c22825d8d7b9237c1d29252dcf3f68f77de600ad18e99d6182d5667"} Jul 17 22:15:52.382668 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:52.382289 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" event={"ID":"357d1923-afdc-4e83-893a-374e855a14e2","Type":"ContainerStarted","Data":"0fed69c60c809e2b4fb540e6adf5ca5f458943342ab64ce0ef8d9bd8bc0c2b8b"} Jul 17 22:15:52.382668 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:52.382318 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" Jul 17 22:15:52.400951 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:52.400900 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" podStartSLOduration=2.002686997 podStartE2EDuration="2.400884s" totalImagesPullingTime="398.197003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:15:52.399525115 +0000 UTC m=+2893.821244366" watchObservedRunningTime="2026-07-17 22:15:52.400884 +0000 UTC m=+2893.822603232" Jul 17 22:15:54.732533 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:54.732494 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:55.196465 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:55.196432 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:55.551554 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:55.551519 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:55.997818 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:55.997724 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:58.241243 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:58.241197 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:58.335287 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:58.335255 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:58.336680 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:58.336656 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq"] Jul 17 22:15:58.340189 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:58.340171 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" Jul 17 22:15:58.342568 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:58.342545 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6fbd36-serving-cert\"" Jul 17 22:15:58.350099 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:58.350065 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq"] Jul 17 22:15:58.393507 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:58.393481 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" Jul 17 22:15:58.407011 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:58.406974 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:58.496900 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:58.496802 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/930c111c-f72b-499e-9902-9870a44316ae-openshift-service-ca\") pod \"maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq\" (UID: \"930c111c-f72b-499e-9902-9870a44316ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" Jul 17 22:15:58.496900 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:58.496890 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/930c111c-f72b-499e-9902-9870a44316ae-maas-api-tls\") pod \"maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq\" (UID: \"930c111c-f72b-499e-9902-9870a44316ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" Jul 17 22:15:58.497177 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:58.496926 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjkjv\" (UniqueName: \"kubernetes.io/projected/930c111c-f72b-499e-9902-9870a44316ae-kube-api-access-gjkjv\") pod \"maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq\" (UID: \"930c111c-f72b-499e-9902-9870a44316ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" Jul 17 22:15:58.598162 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:58.598122 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/930c111c-f72b-499e-9902-9870a44316ae-openshift-service-ca\") pod \"maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq\" (UID: \"930c111c-f72b-499e-9902-9870a44316ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" Jul 17 22:15:58.598469 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:58.598225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/930c111c-f72b-499e-9902-9870a44316ae-maas-api-tls\") pod \"maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq\" (UID: \"930c111c-f72b-499e-9902-9870a44316ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" Jul 17 22:15:58.598469 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:58.598278 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjkjv\" (UniqueName: \"kubernetes.io/projected/930c111c-f72b-499e-9902-9870a44316ae-kube-api-access-gjkjv\") pod \"maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq\" (UID: \"930c111c-f72b-499e-9902-9870a44316ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" Jul 17 22:15:58.598469 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:15:58.598372 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-serving-cert: secret "maas-api-e2e-api-b-6fbd36-serving-cert" not found Jul 17 22:15:58.598677 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:15:58.598483 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/930c111c-f72b-499e-9902-9870a44316ae-maas-api-tls podName:930c111c-f72b-499e-9902-9870a44316ae nodeName:}" failed. No retries permitted until 2026-07-17 22:15:59.098464969 +0000 UTC m=+2900.520184180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/930c111c-f72b-499e-9902-9870a44316ae-maas-api-tls") pod "maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" (UID: "930c111c-f72b-499e-9902-9870a44316ae") : secret "maas-api-e2e-api-b-6fbd36-serving-cert" not found Jul 17 22:15:58.598834 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:58.598816 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/930c111c-f72b-499e-9902-9870a44316ae-openshift-service-ca\") pod \"maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq\" (UID: \"930c111c-f72b-499e-9902-9870a44316ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" Jul 17 22:15:58.609643 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:58.609621 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjkjv\" (UniqueName: \"kubernetes.io/projected/930c111c-f72b-499e-9902-9870a44316ae-kube-api-access-gjkjv\") pod \"maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq\" (UID: \"930c111c-f72b-499e-9902-9870a44316ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" Jul 17 22:15:58.767682 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:58.767649 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:58.892574 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:58.892541 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:58.995822 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:58.995782 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:15:59.102762 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:59.102658 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/930c111c-f72b-499e-9902-9870a44316ae-maas-api-tls\") pod \"maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq\" (UID: \"930c111c-f72b-499e-9902-9870a44316ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" Jul 17 22:15:59.105372 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:59.105340 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/930c111c-f72b-499e-9902-9870a44316ae-maas-api-tls\") pod \"maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq\" (UID: \"930c111c-f72b-499e-9902-9870a44316ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" Jul 17 22:15:59.252920 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:59.252881 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" Jul 17 22:15:59.396758 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:59.396729 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq"] Jul 17 22:15:59.398590 ip-10-0-130-56 kubenswrapper[2580]: W0717 22:15:59.398558 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930c111c_f72b_499e_9902_9870a44316ae.slice/crio-972556b39da4c24a64743f4845e6e2e5431ca2b605dea399698ee82daacff755 WatchSource:0}: Error finding container 972556b39da4c24a64743f4845e6e2e5431ca2b605dea399698ee82daacff755: Status 404 returned error can't find the container with id 972556b39da4c24a64743f4845e6e2e5431ca2b605dea399698ee82daacff755 Jul 17 22:15:59.422118 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:15:59.422088 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" event={"ID":"930c111c-f72b-499e-9902-9870a44316ae","Type":"ContainerStarted","Data":"972556b39da4c24a64743f4845e6e2e5431ca2b605dea399698ee82daacff755"} Jul 17 22:16:00.429774 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:00.429743 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" event={"ID":"930c111c-f72b-499e-9902-9870a44316ae","Type":"ContainerStarted","Data":"14b3250eb418e77090c5e888e53e10cf7a454928aa82704f6c08c2927802e94b"} Jul 17 22:16:00.430205 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:00.429833 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" Jul 17 22:16:00.446992 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:00.446931 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" podStartSLOduration=2.104774102 podStartE2EDuration="2.446915451s" totalImagesPullingTime="342.141349ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:16:00.445475985 +0000 UTC m=+2901.867195239" watchObservedRunningTime="2026-07-17 22:16:00.446915451 +0000 UTC m=+2901.868634683" Jul 17 22:16:02.851166 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:02.851124 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:06.440775 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:06.440747 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" Jul 17 22:16:11.414088 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:11.414052 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:11.510937 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:11.510904 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:19.686438 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:19.686386 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:19.787253 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:19.787216 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:21.332253 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:21.332208 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:21.425145 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:21.425107 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:21.551994 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:21.551956 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw"] Jul 17 22:16:21.552259 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:21.552232 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" podUID="357d1923-afdc-4e83-893a-374e855a14e2" containerName="maas-api" containerID="cri-o://31fb3d9f6c22825d8d7b9237c1d29252dcf3f68f77de600ad18e99d6182d5667" gracePeriod=30 Jul 17 22:16:21.578917 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:21.578887 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:21.805348 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:21.805314 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" Jul 17 22:16:21.910959 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:21.910925 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357d1923-afdc-4e83-893a-374e855a14e2-maas-api-tls\") pod \"357d1923-afdc-4e83-893a-374e855a14e2\" (UID: \"357d1923-afdc-4e83-893a-374e855a14e2\") " Jul 17 22:16:21.910959 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:21.910967 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/357d1923-afdc-4e83-893a-374e855a14e2-openshift-service-ca\") pod \"357d1923-afdc-4e83-893a-374e855a14e2\" (UID: \"357d1923-afdc-4e83-893a-374e855a14e2\") " Jul 17 22:16:21.911191 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:21.910984 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vh8d\" (UniqueName: \"kubernetes.io/projected/357d1923-afdc-4e83-893a-374e855a14e2-kube-api-access-8vh8d\") pod \"357d1923-afdc-4e83-893a-374e855a14e2\" (UID: \"357d1923-afdc-4e83-893a-374e855a14e2\") " Jul 17 22:16:21.911376 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:21.911341 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357d1923-afdc-4e83-893a-374e855a14e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "357d1923-afdc-4e83-893a-374e855a14e2" (UID: "357d1923-afdc-4e83-893a-374e855a14e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:16:21.913262 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:21.913226 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357d1923-afdc-4e83-893a-374e855a14e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "357d1923-afdc-4e83-893a-374e855a14e2" (UID: "357d1923-afdc-4e83-893a-374e855a14e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:16:21.913262 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:21.913247 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357d1923-afdc-4e83-893a-374e855a14e2-kube-api-access-8vh8d" (OuterVolumeSpecName: "kube-api-access-8vh8d") pod "357d1923-afdc-4e83-893a-374e855a14e2" (UID: "357d1923-afdc-4e83-893a-374e855a14e2"). InnerVolumeSpecName "kube-api-access-8vh8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:16:22.011905 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:22.011876 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357d1923-afdc-4e83-893a-374e855a14e2-maas-api-tls\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:16:22.011905 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:22.011905 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/357d1923-afdc-4e83-893a-374e855a14e2-openshift-service-ca\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:16:22.012104 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:22.011914 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vh8d\" (UniqueName: \"kubernetes.io/projected/357d1923-afdc-4e83-893a-374e855a14e2-kube-api-access-8vh8d\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:16:22.548051 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:22.548017 2580 generic.go:356] "Generic (PLEG): container finished" podID="357d1923-afdc-4e83-893a-374e855a14e2" containerID="31fb3d9f6c22825d8d7b9237c1d29252dcf3f68f77de600ad18e99d6182d5667" exitCode=0 Jul 17 22:16:22.548495 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:22.548065 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" Jul 17 22:16:22.548495 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:22.548084 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" event={"ID":"357d1923-afdc-4e83-893a-374e855a14e2","Type":"ContainerDied","Data":"31fb3d9f6c22825d8d7b9237c1d29252dcf3f68f77de600ad18e99d6182d5667"} Jul 17 22:16:22.548495 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:22.548127 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw" event={"ID":"357d1923-afdc-4e83-893a-374e855a14e2","Type":"ContainerDied","Data":"0fed69c60c809e2b4fb540e6adf5ca5f458943342ab64ce0ef8d9bd8bc0c2b8b"} Jul 17 22:16:22.548495 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:22.548148 2580 scope.go:117] "RemoveContainer" containerID="31fb3d9f6c22825d8d7b9237c1d29252dcf3f68f77de600ad18e99d6182d5667" Jul 17 22:16:22.559768 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:22.559563 2580 scope.go:117] "RemoveContainer" containerID="31fb3d9f6c22825d8d7b9237c1d29252dcf3f68f77de600ad18e99d6182d5667" Jul 17 22:16:22.559900 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:16:22.559880 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31fb3d9f6c22825d8d7b9237c1d29252dcf3f68f77de600ad18e99d6182d5667\": container with ID starting with 31fb3d9f6c22825d8d7b9237c1d29252dcf3f68f77de600ad18e99d6182d5667 not found: ID does not exist" containerID="31fb3d9f6c22825d8d7b9237c1d29252dcf3f68f77de600ad18e99d6182d5667" Jul 17 22:16:22.559945 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:22.559908 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31fb3d9f6c22825d8d7b9237c1d29252dcf3f68f77de600ad18e99d6182d5667"} err="failed to get container status \"31fb3d9f6c22825d8d7b9237c1d29252dcf3f68f77de600ad18e99d6182d5667\": rpc error: code = NotFound desc = could not find container \"31fb3d9f6c22825d8d7b9237c1d29252dcf3f68f77de600ad18e99d6182d5667\": container with ID starting with 31fb3d9f6c22825d8d7b9237c1d29252dcf3f68f77de600ad18e99d6182d5667 not found: ID does not exist" Jul 17 22:16:22.574808 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:22.574780 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw"] Jul 17 22:16:22.581042 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:22.581018 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95299c-77589dcbdd-8mhfw"] Jul 17 22:16:23.023671 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:23.023635 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="357d1923-afdc-4e83-893a-374e855a14e2" path="/var/lib/kubelet/pods/357d1923-afdc-4e83-893a-374e855a14e2/volumes" Jul 17 22:16:25.892760 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:25.892728 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:31.697218 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:31.697185 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:31.796843 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:31.796800 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:34.046746 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:34.046707 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:34.149079 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:34.149039 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:36.576907 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:36.576856 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:36.793332 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:36.793294 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:37.448738 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:37.448707 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:37.545181 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:37.545145 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:47.429933 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:47.429892 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq"] Jul 17 22:16:47.430469 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:47.430160 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" podUID="930c111c-f72b-499e-9902-9870a44316ae" containerName="maas-api" containerID="cri-o://14b3250eb418e77090c5e888e53e10cf7a454928aa82704f6c08c2927802e94b" gracePeriod=30 Jul 17 22:16:47.459333 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:47.459303 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:47.678188 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:47.678148 2580 generic.go:356] "Generic (PLEG): container finished" podID="930c111c-f72b-499e-9902-9870a44316ae" containerID="14b3250eb418e77090c5e888e53e10cf7a454928aa82704f6c08c2927802e94b" exitCode=0 Jul 17 22:16:47.678309 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:47.678226 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" event={"ID":"930c111c-f72b-499e-9902-9870a44316ae","Type":"ContainerDied","Data":"14b3250eb418e77090c5e888e53e10cf7a454928aa82704f6c08c2927802e94b"} Jul 17 22:16:47.678309 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:47.678278 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" event={"ID":"930c111c-f72b-499e-9902-9870a44316ae","Type":"ContainerDied","Data":"972556b39da4c24a64743f4845e6e2e5431ca2b605dea399698ee82daacff755"} Jul 17 22:16:47.678309 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:47.678296 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="972556b39da4c24a64743f4845e6e2e5431ca2b605dea399698ee82daacff755" Jul 17 22:16:47.683267 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:47.683215 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" Jul 17 22:16:47.849603 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:47.849563 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/930c111c-f72b-499e-9902-9870a44316ae-openshift-service-ca\") pod \"930c111c-f72b-499e-9902-9870a44316ae\" (UID: \"930c111c-f72b-499e-9902-9870a44316ae\") " Jul 17 22:16:47.849808 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:47.849645 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjkjv\" (UniqueName: \"kubernetes.io/projected/930c111c-f72b-499e-9902-9870a44316ae-kube-api-access-gjkjv\") pod \"930c111c-f72b-499e-9902-9870a44316ae\" (UID: \"930c111c-f72b-499e-9902-9870a44316ae\") " Jul 17 22:16:47.849808 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:47.849674 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/930c111c-f72b-499e-9902-9870a44316ae-maas-api-tls\") pod \"930c111c-f72b-499e-9902-9870a44316ae\" (UID: \"930c111c-f72b-499e-9902-9870a44316ae\") " Jul 17 22:16:47.849968 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:47.849939 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930c111c-f72b-499e-9902-9870a44316ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "930c111c-f72b-499e-9902-9870a44316ae" (UID: "930c111c-f72b-499e-9902-9870a44316ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:16:47.851960 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:47.851924 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930c111c-f72b-499e-9902-9870a44316ae-kube-api-access-gjkjv" (OuterVolumeSpecName: "kube-api-access-gjkjv") pod "930c111c-f72b-499e-9902-9870a44316ae" (UID: "930c111c-f72b-499e-9902-9870a44316ae"). InnerVolumeSpecName "kube-api-access-gjkjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:16:47.851960 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:47.851944 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/930c111c-f72b-499e-9902-9870a44316ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "930c111c-f72b-499e-9902-9870a44316ae" (UID: "930c111c-f72b-499e-9902-9870a44316ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:16:47.951299 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:47.951212 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/930c111c-f72b-499e-9902-9870a44316ae-openshift-service-ca\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:16:47.951299 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:47.951244 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjkjv\" (UniqueName: \"kubernetes.io/projected/930c111c-f72b-499e-9902-9870a44316ae-kube-api-access-gjkjv\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:16:47.951299 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:47.951254 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/930c111c-f72b-499e-9902-9870a44316ae-maas-api-tls\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:16:48.683101 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:48.683069 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq" Jul 17 22:16:48.704573 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:48.704538 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq"] Jul 17 22:16:48.707706 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:48.707682 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6fbd36-84fb769567-8wzwq"] Jul 17 22:16:49.024263 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:49.024230 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="930c111c-f72b-499e-9902-9870a44316ae" path="/var/lib/kubelet/pods/930c111c-f72b-499e-9902-9870a44316ae/volumes" Jul 17 22:16:51.996203 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:51.996166 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:57.652205 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:57.652165 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:16:57.712041 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:16:57.712004 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:02.691880 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:02.691803 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:02.919148 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:02.919108 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:03.254203 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:03.254172 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:03.578649 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:03.578616 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:04.066271 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:04.066232 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:06.653226 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:06.653190 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:06.832248 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:06.832213 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:06.834932 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:06.834905 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5"] Jul 17 22:17:06.835269 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:06.835256 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="930c111c-f72b-499e-9902-9870a44316ae" containerName="maas-api" Jul 17 22:17:06.835269 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:06.835269 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="930c111c-f72b-499e-9902-9870a44316ae" containerName="maas-api" Jul 17 22:17:06.835354 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:06.835288 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357d1923-afdc-4e83-893a-374e855a14e2" containerName="maas-api" Jul 17 22:17:06.835354 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:06.835294 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="357d1923-afdc-4e83-893a-374e855a14e2" containerName="maas-api" Jul 17 22:17:06.835354 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:06.835347 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="357d1923-afdc-4e83-893a-374e855a14e2" containerName="maas-api" Jul 17 22:17:06.835480 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:06.835360 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="930c111c-f72b-499e-9902-9870a44316ae" containerName="maas-api" Jul 17 22:17:06.838564 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:06.838547 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" Jul 17 22:17:06.841412 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:06.841370 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-8e6a3a-serving-cert\"" Jul 17 22:17:06.854972 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:06.854948 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5"] Jul 17 22:17:06.922303 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:06.922206 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d05686e-37e8-4b0d-bc5c-9a7d6e97399d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5\" (UID: \"3d05686e-37e8-4b0d-bc5c-9a7d6e97399d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" Jul 17 22:17:06.922505 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:06.922304 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d05686e-37e8-4b0d-bc5c-9a7d6e97399d-maas-api-tls\") pod \"maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5\" (UID: \"3d05686e-37e8-4b0d-bc5c-9a7d6e97399d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" Jul 17 22:17:06.922505 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:06.922357 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phrbb\" (UniqueName: \"kubernetes.io/projected/3d05686e-37e8-4b0d-bc5c-9a7d6e97399d-kube-api-access-phrbb\") pod \"maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5\" (UID: \"3d05686e-37e8-4b0d-bc5c-9a7d6e97399d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" Jul 17 22:17:06.940636 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:06.940604 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:07.023636 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:07.023601 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d05686e-37e8-4b0d-bc5c-9a7d6e97399d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5\" (UID: \"3d05686e-37e8-4b0d-bc5c-9a7d6e97399d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" Jul 17 22:17:07.023838 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:07.023673 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d05686e-37e8-4b0d-bc5c-9a7d6e97399d-maas-api-tls\") pod \"maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5\" (UID: \"3d05686e-37e8-4b0d-bc5c-9a7d6e97399d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" Jul 17 22:17:07.023838 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:07.023731 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phrbb\" (UniqueName: \"kubernetes.io/projected/3d05686e-37e8-4b0d-bc5c-9a7d6e97399d-kube-api-access-phrbb\") pod \"maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5\" (UID: \"3d05686e-37e8-4b0d-bc5c-9a7d6e97399d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" Jul 17 22:17:07.024266 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:07.024246 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d05686e-37e8-4b0d-bc5c-9a7d6e97399d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5\" (UID: \"3d05686e-37e8-4b0d-bc5c-9a7d6e97399d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" Jul 17 22:17:07.026534 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:07.026513 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d05686e-37e8-4b0d-bc5c-9a7d6e97399d-maas-api-tls\") pod \"maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5\" (UID: \"3d05686e-37e8-4b0d-bc5c-9a7d6e97399d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" Jul 17 22:17:07.033639 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:07.033611 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phrbb\" (UniqueName: \"kubernetes.io/projected/3d05686e-37e8-4b0d-bc5c-9a7d6e97399d-kube-api-access-phrbb\") pod \"maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5\" (UID: \"3d05686e-37e8-4b0d-bc5c-9a7d6e97399d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" Jul 17 22:17:07.040350 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:07.040314 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:07.148550 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:07.148507 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" Jul 17 22:17:07.292941 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:07.292915 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5"] Jul 17 22:17:07.295101 ip-10-0-130-56 kubenswrapper[2580]: W0717 22:17:07.295065 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d05686e_37e8_4b0d_bc5c_9a7d6e97399d.slice/crio-68a3eb7c5a2253d1405ae1da7586893b6ca54d1197b2f98766d0e0dc094d4152 WatchSource:0}: Error finding container 68a3eb7c5a2253d1405ae1da7586893b6ca54d1197b2f98766d0e0dc094d4152: Status 404 returned error can't find the container with id 68a3eb7c5a2253d1405ae1da7586893b6ca54d1197b2f98766d0e0dc094d4152 Jul 17 22:17:07.451730 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:07.451640 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:07.555837 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:07.555791 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:07.786671 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:07.786645 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" event={"ID":"3d05686e-37e8-4b0d-bc5c-9a7d6e97399d","Type":"ContainerStarted","Data":"68a3eb7c5a2253d1405ae1da7586893b6ca54d1197b2f98766d0e0dc094d4152"} Jul 17 22:17:08.793646 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:08.793613 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" event={"ID":"3d05686e-37e8-4b0d-bc5c-9a7d6e97399d","Type":"ContainerStarted","Data":"dbd916a94bf366ca9decfb262e8cc3e77f6362443721407c61e7000e084bb955"} Jul 17 22:17:08.794099 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:08.793678 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" Jul 17 22:17:08.810653 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:08.810607 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" podStartSLOduration=2.3553649979999998 podStartE2EDuration="2.810592628s" totalImagesPullingTime="455.22763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:17:08.809005634 +0000 UTC m=+2970.230724868" watchObservedRunningTime="2026-07-17 22:17:08.810592628 +0000 UTC m=+2970.232311861" Jul 17 22:17:09.202748 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:09.202634 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:11.127717 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:11.127671 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:11.457852 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:11.457769 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:11.793121 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:11.793086 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:12.032207 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:12.032170 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:14.781601 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:14.781562 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:14.804697 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:14.804670 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" Jul 17 22:17:14.880303 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:14.880269 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:15.015097 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:15.015053 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m"] Jul 17 22:17:15.019434 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:15.019404 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" Jul 17 22:17:15.022096 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:15.022073 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-5401e5-serving-cert\"" Jul 17 22:17:15.033216 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:15.033141 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m"] Jul 17 22:17:15.044934 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:15.044879 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:15.097064 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:15.097025 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj5c2\" (UniqueName: \"kubernetes.io/projected/713c23d7-0f21-4eea-b69e-71d6ea4c476f-kube-api-access-lj5c2\") pod \"maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m\" (UID: \"713c23d7-0f21-4eea-b69e-71d6ea4c476f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" Jul 17 22:17:15.097216 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:15.097073 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/713c23d7-0f21-4eea-b69e-71d6ea4c476f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m\" (UID: \"713c23d7-0f21-4eea-b69e-71d6ea4c476f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" Jul 17 22:17:15.097277 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:15.097205 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/713c23d7-0f21-4eea-b69e-71d6ea4c476f-maas-api-tls\") pod \"maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m\" (UID: \"713c23d7-0f21-4eea-b69e-71d6ea4c476f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" Jul 17 22:17:15.198335 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:15.198295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/713c23d7-0f21-4eea-b69e-71d6ea4c476f-maas-api-tls\") pod \"maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m\" (UID: \"713c23d7-0f21-4eea-b69e-71d6ea4c476f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" Jul 17 22:17:15.198555 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:15.198349 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj5c2\" (UniqueName: \"kubernetes.io/projected/713c23d7-0f21-4eea-b69e-71d6ea4c476f-kube-api-access-lj5c2\") pod \"maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m\" (UID: \"713c23d7-0f21-4eea-b69e-71d6ea4c476f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" Jul 17 22:17:15.198555 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:15.198376 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/713c23d7-0f21-4eea-b69e-71d6ea4c476f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m\" (UID: \"713c23d7-0f21-4eea-b69e-71d6ea4c476f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" Jul 17 22:17:15.198555 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:17:15.198496 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-serving-cert: secret "maas-api-e2e-shared-b-5401e5-serving-cert" not found Jul 17 22:17:15.198718 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:17:15.198594 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/713c23d7-0f21-4eea-b69e-71d6ea4c476f-maas-api-tls podName:713c23d7-0f21-4eea-b69e-71d6ea4c476f nodeName:}" failed. No retries permitted until 2026-07-17 22:17:15.698571644 +0000 UTC m=+2977.120290861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/713c23d7-0f21-4eea-b69e-71d6ea4c476f-maas-api-tls") pod "maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" (UID: "713c23d7-0f21-4eea-b69e-71d6ea4c476f") : secret "maas-api-e2e-shared-b-5401e5-serving-cert" not found Jul 17 22:17:15.199006 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:15.198988 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/713c23d7-0f21-4eea-b69e-71d6ea4c476f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m\" (UID: \"713c23d7-0f21-4eea-b69e-71d6ea4c476f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" Jul 17 22:17:15.207303 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:15.207281 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj5c2\" (UniqueName: \"kubernetes.io/projected/713c23d7-0f21-4eea-b69e-71d6ea4c476f-kube-api-access-lj5c2\") pod \"maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m\" (UID: \"713c23d7-0f21-4eea-b69e-71d6ea4c476f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" Jul 17 22:17:15.523307 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:15.523269 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:15.619495 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:15.619460 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:15.703785 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:15.703748 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/713c23d7-0f21-4eea-b69e-71d6ea4c476f-maas-api-tls\") pod \"maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m\" (UID: \"713c23d7-0f21-4eea-b69e-71d6ea4c476f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" Jul 17 22:17:15.706523 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:15.706488 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/713c23d7-0f21-4eea-b69e-71d6ea4c476f-maas-api-tls\") pod \"maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m\" (UID: \"713c23d7-0f21-4eea-b69e-71d6ea4c476f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" Jul 17 22:17:15.732492 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:15.732460 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:15.934341 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:15.934244 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" Jul 17 22:17:16.073521 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:16.073493 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m"] Jul 17 22:17:16.075617 ip-10-0-130-56 kubenswrapper[2580]: W0717 22:17:16.075584 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713c23d7_0f21_4eea_b69e_71d6ea4c476f.slice/crio-efe7b9919d122d1efd34db131f53925b7e24b161ff5fd7c5dca5cfdb4838d0be WatchSource:0}: Error finding container efe7b9919d122d1efd34db131f53925b7e24b161ff5fd7c5dca5cfdb4838d0be: Status 404 returned error can't find the container with id efe7b9919d122d1efd34db131f53925b7e24b161ff5fd7c5dca5cfdb4838d0be Jul 17 22:17:16.839162 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:16.839120 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" event={"ID":"713c23d7-0f21-4eea-b69e-71d6ea4c476f","Type":"ContainerStarted","Data":"6c94a606a6ba3eda8fb5ed10e5607648f609b1618578a33679e15649ba329f44"} Jul 17 22:17:16.839335 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:16.839181 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" event={"ID":"713c23d7-0f21-4eea-b69e-71d6ea4c476f","Type":"ContainerStarted","Data":"efe7b9919d122d1efd34db131f53925b7e24b161ff5fd7c5dca5cfdb4838d0be"} Jul 17 22:17:16.839335 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:16.839225 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" Jul 17 22:17:16.855013 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:16.854965 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" podStartSLOduration=2.532003295 podStartE2EDuration="2.854949045s" totalImagesPullingTime="322.94575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:17:16.853579355 +0000 UTC m=+2978.275298617" watchObservedRunningTime="2026-07-17 22:17:16.854949045 +0000 UTC m=+2978.276668278" Jul 17 22:17:19.179408 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:19.179356 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:22.851062 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:22.851033 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" Jul 17 22:17:26.532586 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:26.532545 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:29.184768 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:29.184738 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:29.281676 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:29.281635 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:36.248505 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:36.248472 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:36.350817 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:36.350781 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:37.642659 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:37.642621 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:17:39.244066 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:39.244036 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 22:17:39.256173 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:39.256152 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 22:17:42.885298 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:17:42.885264 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:19:01.335160 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:01.335077 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jx2kj"] Jul 17 22:19:01.338054 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:01.338037 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jx2kj" Jul 17 22:19:01.340770 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:01.340748 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-t2xg8\"" Jul 17 22:19:01.348081 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:01.348058 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jx2kj"] Jul 17 22:19:01.476138 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:01.476100 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhs59\" (UniqueName: \"kubernetes.io/projected/f99762b2-7251-459f-a247-9d42faad68df-kube-api-access-dhs59\") pod \"kuadrant-operator-catalog-jx2kj\" (UID: \"f99762b2-7251-459f-a247-9d42faad68df\") " pod="kuadrant-system/kuadrant-operator-catalog-jx2kj" Jul 17 22:19:01.577629 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:01.577594 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhs59\" (UniqueName: \"kubernetes.io/projected/f99762b2-7251-459f-a247-9d42faad68df-kube-api-access-dhs59\") pod \"kuadrant-operator-catalog-jx2kj\" (UID: \"f99762b2-7251-459f-a247-9d42faad68df\") " pod="kuadrant-system/kuadrant-operator-catalog-jx2kj" Jul 17 22:19:01.586758 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:01.586685 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhs59\" (UniqueName: \"kubernetes.io/projected/f99762b2-7251-459f-a247-9d42faad68df-kube-api-access-dhs59\") pod \"kuadrant-operator-catalog-jx2kj\" (UID: \"f99762b2-7251-459f-a247-9d42faad68df\") " pod="kuadrant-system/kuadrant-operator-catalog-jx2kj" Jul 17 22:19:01.649049 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:01.649014 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jx2kj" Jul 17 22:19:01.785322 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:01.785299 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jx2kj"] Jul 17 22:19:01.787455 ip-10-0-130-56 kubenswrapper[2580]: W0717 22:19:01.787426 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99762b2_7251_459f_a247_9d42faad68df.slice/crio-f07cf3493644f8cd15324ffc098d1e3712f91aee31a83da5628f923298101fef WatchSource:0}: Error finding container f07cf3493644f8cd15324ffc098d1e3712f91aee31a83da5628f923298101fef: Status 404 returned error can't find the container with id f07cf3493644f8cd15324ffc098d1e3712f91aee31a83da5628f923298101fef Jul 17 22:19:02.414887 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:02.414852 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jx2kj" event={"ID":"f99762b2-7251-459f-a247-9d42faad68df","Type":"ContainerStarted","Data":"f07cf3493644f8cd15324ffc098d1e3712f91aee31a83da5628f923298101fef"} Jul 17 22:19:03.425088 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:03.425047 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jx2kj" event={"ID":"f99762b2-7251-459f-a247-9d42faad68df","Type":"ContainerStarted","Data":"d133084eecce51d4ae1bd4dda434921e35f32b933dde599b728dfcb199172f12"} Jul 17 22:19:03.443502 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:03.443438 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-jx2kj" podStartSLOduration=0.946386538 podStartE2EDuration="2.443415328s" totalImagesPullingTime="1.49702879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:19:03.441916997 +0000 UTC m=+3084.863636231" watchObservedRunningTime="2026-07-17 22:19:03.443415328 +0000 UTC m=+3084.865134557" Jul 17 22:19:11.649620 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:11.649591 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-jx2kj" Jul 17 22:19:11.650082 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:11.649649 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-jx2kj" Jul 17 22:19:11.673544 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:11.673515 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-jx2kj" Jul 17 22:19:12.502321 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:12.502274 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-jx2kj" Jul 17 22:19:12.535818 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:12.535782 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jx2kj"] Jul 17 22:19:14.490597 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:14.490549 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-jx2kj" podUID="f99762b2-7251-459f-a247-9d42faad68df" containerName="registry-server" containerID="cri-o://d133084eecce51d4ae1bd4dda434921e35f32b933dde599b728dfcb199172f12" gracePeriod=2 Jul 17 22:19:14.742512 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:14.742435 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jx2kj" Jul 17 22:19:14.799726 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:14.799697 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhs59\" (UniqueName: \"kubernetes.io/projected/f99762b2-7251-459f-a247-9d42faad68df-kube-api-access-dhs59\") pod \"f99762b2-7251-459f-a247-9d42faad68df\" (UID: \"f99762b2-7251-459f-a247-9d42faad68df\") " Jul 17 22:19:14.801952 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:14.801920 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99762b2-7251-459f-a247-9d42faad68df-kube-api-access-dhs59" (OuterVolumeSpecName: "kube-api-access-dhs59") pod "f99762b2-7251-459f-a247-9d42faad68df" (UID: "f99762b2-7251-459f-a247-9d42faad68df"). InnerVolumeSpecName "kube-api-access-dhs59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:19:14.901005 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:14.900966 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhs59\" (UniqueName: \"kubernetes.io/projected/f99762b2-7251-459f-a247-9d42faad68df-kube-api-access-dhs59\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:19:15.497250 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:15.497167 2580 generic.go:356] "Generic (PLEG): container finished" podID="f99762b2-7251-459f-a247-9d42faad68df" containerID="d133084eecce51d4ae1bd4dda434921e35f32b933dde599b728dfcb199172f12" exitCode=0 Jul 17 22:19:15.497250 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:15.497215 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jx2kj" Jul 17 22:19:15.497250 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:15.497240 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jx2kj" event={"ID":"f99762b2-7251-459f-a247-9d42faad68df","Type":"ContainerDied","Data":"d133084eecce51d4ae1bd4dda434921e35f32b933dde599b728dfcb199172f12"} Jul 17 22:19:15.497736 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:15.497278 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jx2kj" event={"ID":"f99762b2-7251-459f-a247-9d42faad68df","Type":"ContainerDied","Data":"f07cf3493644f8cd15324ffc098d1e3712f91aee31a83da5628f923298101fef"} Jul 17 22:19:15.497736 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:15.497297 2580 scope.go:117] "RemoveContainer" containerID="d133084eecce51d4ae1bd4dda434921e35f32b933dde599b728dfcb199172f12" Jul 17 22:19:15.507033 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:15.507012 2580 scope.go:117] "RemoveContainer" containerID="d133084eecce51d4ae1bd4dda434921e35f32b933dde599b728dfcb199172f12" Jul 17 22:19:15.507315 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:19:15.507293 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d133084eecce51d4ae1bd4dda434921e35f32b933dde599b728dfcb199172f12\": container with ID starting with d133084eecce51d4ae1bd4dda434921e35f32b933dde599b728dfcb199172f12 not found: ID does not exist" containerID="d133084eecce51d4ae1bd4dda434921e35f32b933dde599b728dfcb199172f12" Jul 17 22:19:15.507363 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:15.507325 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d133084eecce51d4ae1bd4dda434921e35f32b933dde599b728dfcb199172f12"} err="failed to get container status \"d133084eecce51d4ae1bd4dda434921e35f32b933dde599b728dfcb199172f12\": rpc error: code = NotFound desc = could not find container \"d133084eecce51d4ae1bd4dda434921e35f32b933dde599b728dfcb199172f12\": container with ID starting with d133084eecce51d4ae1bd4dda434921e35f32b933dde599b728dfcb199172f12 not found: ID does not exist" Jul 17 22:19:15.515034 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:15.515009 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jx2kj"] Jul 17 22:19:15.519073 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:15.519053 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jx2kj"] Jul 17 22:19:17.024554 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:17.024517 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f99762b2-7251-459f-a247-9d42faad68df" path="/var/lib/kubelet/pods/f99762b2-7251-459f-a247-9d42faad68df/volumes" Jul 17 22:19:41.611428 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:41.611369 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:19:42.801004 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:42.800970 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:19:46.717167 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:46.717130 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:19:47.715604 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:47.715573 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:19:53.232825 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:19:53.232793 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:20:17.233900 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:20:17.233864 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:20:23.657829 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:20:23.657779 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:21:03.730882 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:21:03.730846 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:21:04.800819 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:21:04.800788 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:21:34.840734 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:21:34.840630 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:21:34.941735 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:21:34.941688 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:21:35.041001 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:21:35.040966 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:21:56.143033 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:21:56.142998 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:22:21.259695 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:21.259644 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:22:21.360154 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:21.360121 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:22:29.053688 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:29.053648 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:22:30.004501 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:30.004463 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:22:30.104020 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:30.103989 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:22:31.702710 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:31.702672 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:22:31.759359 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:31.759315 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:22:37.079710 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:37.079677 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:22:37.178238 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:37.178184 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:22:39.281613 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:39.281584 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 22:22:39.293568 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:39.293548 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rhkgr_ba80c237-8ba4-4bdc-9592-c8fb5bf8d912/ovn-acl-logging/0.log" Jul 17 22:22:39.418479 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:39.418445 2580 scope.go:117] "RemoveContainer" containerID="14b3250eb418e77090c5e888e53e10cf7a454928aa82704f6c08c2927802e94b" Jul 17 22:22:49.857718 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:49.857673 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:22:50.998159 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:50.998110 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:22:51.937576 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:51.937543 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7321-139979028326960"] Jul 17 22:22:51.937927 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:51.937913 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f99762b2-7251-459f-a247-9d42faad68df" containerName="registry-server" Jul 17 22:22:51.937978 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:51.937928 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f99762b2-7251-459f-a247-9d42faad68df" containerName="registry-server" Jul 17 22:22:51.938023 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:51.937994 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f99762b2-7251-459f-a247-9d42faad68df" containerName="registry-server" Jul 17 22:22:51.941038 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:51.941022 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979028326960" Jul 17 22:22:51.943772 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:51.943745 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gm6bq\"" Jul 17 22:22:51.946788 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:51.946761 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7321-139979028326960"] Jul 17 22:22:52.027479 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:52.027443 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hdz5\" (UniqueName: \"kubernetes.io/projected/9a7f8090-43ad-4085-88f3-71d56693f474-kube-api-access-9hdz5\") pod \"test-curl-7321-139979028326960\" (UID: \"9a7f8090-43ad-4085-88f3-71d56693f474\") " pod="opendatahub/test-curl-7321-139979028326960" Jul 17 22:22:52.128929 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:52.128890 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdz5\" (UniqueName: \"kubernetes.io/projected/9a7f8090-43ad-4085-88f3-71d56693f474-kube-api-access-9hdz5\") pod \"test-curl-7321-139979028326960\" (UID: \"9a7f8090-43ad-4085-88f3-71d56693f474\") " pod="opendatahub/test-curl-7321-139979028326960" Jul 17 22:22:52.137550 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:52.137521 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hdz5\" (UniqueName: \"kubernetes.io/projected/9a7f8090-43ad-4085-88f3-71d56693f474-kube-api-access-9hdz5\") pod \"test-curl-7321-139979028326960\" (UID: \"9a7f8090-43ad-4085-88f3-71d56693f474\") " pod="opendatahub/test-curl-7321-139979028326960" Jul 17 22:22:52.251986 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:52.251893 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979028326960" Jul 17 22:22:52.389086 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:52.389057 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7321-139979028326960"] Jul 17 22:22:52.391224 ip-10-0-130-56 kubenswrapper[2580]: W0717 22:22:52.391187 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a7f8090_43ad_4085_88f3_71d56693f474.slice/crio-f3a4f58fbe8b19e555f076008edcd2ce8d0f8cf80f1ba04c5608e265eb4f0180 WatchSource:0}: Error finding container f3a4f58fbe8b19e555f076008edcd2ce8d0f8cf80f1ba04c5608e265eb4f0180: Status 404 returned error can't find the container with id f3a4f58fbe8b19e555f076008edcd2ce8d0f8cf80f1ba04c5608e265eb4f0180 Jul 17 22:22:52.392663 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:52.392643 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 22:22:52.681446 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:52.681414 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7321-139979028326960" event={"ID":"9a7f8090-43ad-4085-88f3-71d56693f474","Type":"ContainerStarted","Data":"f3a4f58fbe8b19e555f076008edcd2ce8d0f8cf80f1ba04c5608e265eb4f0180"} Jul 17 22:22:53.688090 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:53.688058 2580 generic.go:356] "Generic (PLEG): container finished" podID="9a7f8090-43ad-4085-88f3-71d56693f474" containerID="abb191b308b19547f23b8704123890149d06883876b1830283b95221515ba878" exitCode=0 Jul 17 22:22:53.688090 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:53.688093 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7321-139979028326960" event={"ID":"9a7f8090-43ad-4085-88f3-71d56693f474","Type":"ContainerDied","Data":"abb191b308b19547f23b8704123890149d06883876b1830283b95221515ba878"} Jul 17 22:22:54.827668 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:54.827639 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979028326960" Jul 17 22:22:54.851408 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:54.851358 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hdz5\" (UniqueName: \"kubernetes.io/projected/9a7f8090-43ad-4085-88f3-71d56693f474-kube-api-access-9hdz5\") pod \"9a7f8090-43ad-4085-88f3-71d56693f474\" (UID: \"9a7f8090-43ad-4085-88f3-71d56693f474\") " Jul 17 22:22:54.851599 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:54.851544 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7321-139979028326960_9a7f8090-43ad-4085-88f3-71d56693f474/test-curl-7321-139979028326960/0.log" Jul 17 22:22:54.854089 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:54.854060 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7f8090-43ad-4085-88f3-71d56693f474-kube-api-access-9hdz5" (OuterVolumeSpecName: "kube-api-access-9hdz5") pod "9a7f8090-43ad-4085-88f3-71d56693f474" (UID: "9a7f8090-43ad-4085-88f3-71d56693f474"). InnerVolumeSpecName "kube-api-access-9hdz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:22:54.879885 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:54.879854 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7321-139979028326960"] Jul 17 22:22:54.883917 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:54.883889 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7321-139979028326960"] Jul 17 22:22:54.953253 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:54.953140 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hdz5\" (UniqueName: \"kubernetes.io/projected/9a7f8090-43ad-4085-88f3-71d56693f474-kube-api-access-9hdz5\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:22:55.007504 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:55.007466 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7321-139979028251568"] Jul 17 22:22:55.007891 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:55.007878 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a7f8090-43ad-4085-88f3-71d56693f474" containerName="test-curl-7321-139979028326960" Jul 17 22:22:55.007937 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:55.007893 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a7f8090-43ad-4085-88f3-71d56693f474" containerName="test-curl-7321-139979028326960" Jul 17 22:22:55.007973 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:55.007968 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a7f8090-43ad-4085-88f3-71d56693f474" containerName="test-curl-7321-139979028326960" Jul 17 22:22:55.011073 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:55.011055 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979028251568" Jul 17 22:22:55.024777 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:55.024743 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a7f8090-43ad-4085-88f3-71d56693f474" path="/var/lib/kubelet/pods/9a7f8090-43ad-4085-88f3-71d56693f474/volumes" Jul 17 22:22:55.025095 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:55.025082 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7321-139979028251568"] Jul 17 22:22:55.053928 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:55.053892 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5lvt\" (UniqueName: \"kubernetes.io/projected/bfd04e9c-92a3-4feb-8ffe-04f15f29c60d-kube-api-access-t5lvt\") pod \"test-curl-7321-139979028251568\" (UID: \"bfd04e9c-92a3-4feb-8ffe-04f15f29c60d\") " pod="opendatahub/test-curl-7321-139979028251568" Jul 17 22:22:55.154994 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:55.154963 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5lvt\" (UniqueName: \"kubernetes.io/projected/bfd04e9c-92a3-4feb-8ffe-04f15f29c60d-kube-api-access-t5lvt\") pod \"test-curl-7321-139979028251568\" (UID: \"bfd04e9c-92a3-4feb-8ffe-04f15f29c60d\") " pod="opendatahub/test-curl-7321-139979028251568" Jul 17 22:22:55.163343 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:55.163308 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5lvt\" (UniqueName: \"kubernetes.io/projected/bfd04e9c-92a3-4feb-8ffe-04f15f29c60d-kube-api-access-t5lvt\") pod \"test-curl-7321-139979028251568\" (UID: \"bfd04e9c-92a3-4feb-8ffe-04f15f29c60d\") " pod="opendatahub/test-curl-7321-139979028251568" Jul 17 22:22:55.321739 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:55.321704 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979028251568" Jul 17 22:22:55.459910 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:55.459889 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7321-139979028251568"] Jul 17 22:22:55.462070 ip-10-0-130-56 kubenswrapper[2580]: W0717 22:22:55.462040 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfd04e9c_92a3_4feb_8ffe_04f15f29c60d.slice/crio-c94fd9ce66d3d58e437da91524d4f90b55b86066b4a31b4e53f1bdcab27794ec WatchSource:0}: Error finding container c94fd9ce66d3d58e437da91524d4f90b55b86066b4a31b4e53f1bdcab27794ec: Status 404 returned error can't find the container with id c94fd9ce66d3d58e437da91524d4f90b55b86066b4a31b4e53f1bdcab27794ec Jul 17 22:22:55.699850 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:55.699765 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979028326960" Jul 17 22:22:55.699850 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:55.699779 2580 scope.go:117] "RemoveContainer" containerID="abb191b308b19547f23b8704123890149d06883876b1830283b95221515ba878" Jul 17 22:22:55.701153 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:55.701131 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7321-139979028251568" event={"ID":"bfd04e9c-92a3-4feb-8ffe-04f15f29c60d","Type":"ContainerStarted","Data":"c94fd9ce66d3d58e437da91524d4f90b55b86066b4a31b4e53f1bdcab27794ec"} Jul 17 22:22:56.708833 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:56.708798 2580 generic.go:356] "Generic (PLEG): container finished" podID="bfd04e9c-92a3-4feb-8ffe-04f15f29c60d" containerID="097526c168547dabae71c3434673516402f629b17b618fafe78508f6ecf893eb" exitCode=0 Jul 17 22:22:56.709254 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:56.708896 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7321-139979028251568" event={"ID":"bfd04e9c-92a3-4feb-8ffe-04f15f29c60d","Type":"ContainerDied","Data":"097526c168547dabae71c3434673516402f629b17b618fafe78508f6ecf893eb"} Jul 17 22:22:57.852691 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:57.852663 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979028251568" Jul 17 22:22:57.873171 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:57.873146 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7321-139979028251568_bfd04e9c-92a3-4feb-8ffe-04f15f29c60d/test-curl-7321-139979028251568/0.log" Jul 17 22:22:57.899049 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:57.899013 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7321-139979028251568"] Jul 17 22:22:57.901258 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:57.901234 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7321-139979028251568"] Jul 17 22:22:57.978302 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:57.978212 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5lvt\" (UniqueName: \"kubernetes.io/projected/bfd04e9c-92a3-4feb-8ffe-04f15f29c60d-kube-api-access-t5lvt\") pod \"bfd04e9c-92a3-4feb-8ffe-04f15f29c60d\" (UID: \"bfd04e9c-92a3-4feb-8ffe-04f15f29c60d\") " Jul 17 22:22:57.980528 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:57.980501 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd04e9c-92a3-4feb-8ffe-04f15f29c60d-kube-api-access-t5lvt" (OuterVolumeSpecName: "kube-api-access-t5lvt") pod "bfd04e9c-92a3-4feb-8ffe-04f15f29c60d" (UID: "bfd04e9c-92a3-4feb-8ffe-04f15f29c60d"). InnerVolumeSpecName "kube-api-access-t5lvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:22:58.067359 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:58.067323 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7321-139979024659776"] Jul 17 22:22:58.067735 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:58.067720 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfd04e9c-92a3-4feb-8ffe-04f15f29c60d" containerName="test-curl-7321-139979028251568" Jul 17 22:22:58.067778 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:58.067737 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfd04e9c-92a3-4feb-8ffe-04f15f29c60d" containerName="test-curl-7321-139979028251568" Jul 17 22:22:58.067825 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:58.067816 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfd04e9c-92a3-4feb-8ffe-04f15f29c60d" containerName="test-curl-7321-139979028251568" Jul 17 22:22:58.070862 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:58.070847 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979024659776" Jul 17 22:22:58.074550 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:58.074521 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7321-139979024659776"] Jul 17 22:22:58.079131 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:58.079105 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5lvt\" (UniqueName: \"kubernetes.io/projected/bfd04e9c-92a3-4feb-8ffe-04f15f29c60d-kube-api-access-t5lvt\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:22:58.180547 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:58.180502 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q8pf\" (UniqueName: \"kubernetes.io/projected/d4265121-e1e6-4fa6-963b-2bd1b3cc432c-kube-api-access-9q8pf\") pod \"test-curl-7321-139979024659776\" (UID: \"d4265121-e1e6-4fa6-963b-2bd1b3cc432c\") " pod="opendatahub/test-curl-7321-139979024659776" Jul 17 22:22:58.281782 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:58.281750 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q8pf\" (UniqueName: \"kubernetes.io/projected/d4265121-e1e6-4fa6-963b-2bd1b3cc432c-kube-api-access-9q8pf\") pod \"test-curl-7321-139979024659776\" (UID: \"d4265121-e1e6-4fa6-963b-2bd1b3cc432c\") " pod="opendatahub/test-curl-7321-139979024659776" Jul 17 22:22:58.289550 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:58.289523 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q8pf\" (UniqueName: \"kubernetes.io/projected/d4265121-e1e6-4fa6-963b-2bd1b3cc432c-kube-api-access-9q8pf\") pod \"test-curl-7321-139979024659776\" (UID: \"d4265121-e1e6-4fa6-963b-2bd1b3cc432c\") " pod="opendatahub/test-curl-7321-139979024659776" Jul 17 22:22:58.382529 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:58.382496 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979024659776" Jul 17 22:22:58.527931 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:58.527905 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7321-139979024659776"] Jul 17 22:22:58.529978 ip-10-0-130-56 kubenswrapper[2580]: W0717 22:22:58.529947 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4265121_e1e6_4fa6_963b_2bd1b3cc432c.slice/crio-4b56fc725f61c4723b212792580913f73cfbcf6ab0ff47ed7745f3de9d2429f6 WatchSource:0}: Error finding container 4b56fc725f61c4723b212792580913f73cfbcf6ab0ff47ed7745f3de9d2429f6: Status 404 returned error can't find the container with id 4b56fc725f61c4723b212792580913f73cfbcf6ab0ff47ed7745f3de9d2429f6 Jul 17 22:22:58.721988 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:58.721949 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7321-139979024659776" event={"ID":"d4265121-e1e6-4fa6-963b-2bd1b3cc432c","Type":"ContainerStarted","Data":"4b56fc725f61c4723b212792580913f73cfbcf6ab0ff47ed7745f3de9d2429f6"} Jul 17 22:22:58.723341 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:58.723316 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979028251568" Jul 17 22:22:58.723494 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:58.723317 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c94fd9ce66d3d58e437da91524d4f90b55b86066b4a31b4e53f1bdcab27794ec" Jul 17 22:22:58.734159 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:22:58.734125 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7321-139979028251568\" is forbidden: User \"system:node:ip-10-0-130-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-56.ec2.internal' and this object" podUID="bfd04e9c-92a3-4feb-8ffe-04f15f29c60d" pod="opendatahub/test-curl-7321-139979028251568" Jul 17 22:22:59.023857 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:22:59.023802 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7321-139979028251568\" is forbidden: User \"system:node:ip-10-0-130-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-56.ec2.internal' and this object" podUID="bfd04e9c-92a3-4feb-8ffe-04f15f29c60d" pod="opendatahub/test-curl-7321-139979028251568" Jul 17 22:22:59.025065 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:59.025044 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfd04e9c-92a3-4feb-8ffe-04f15f29c60d" path="/var/lib/kubelet/pods/bfd04e9c-92a3-4feb-8ffe-04f15f29c60d/volumes" Jul 17 22:22:59.730732 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:59.730654 2580 generic.go:356] "Generic (PLEG): container finished" podID="d4265121-e1e6-4fa6-963b-2bd1b3cc432c" containerID="4bf55020d45e20635f826b8d1b55fc6937d310a50317dcf312c64073ef0dba60" exitCode=0 Jul 17 22:22:59.730732 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:22:59.730701 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7321-139979024659776" event={"ID":"d4265121-e1e6-4fa6-963b-2bd1b3cc432c","Type":"ContainerDied","Data":"4bf55020d45e20635f826b8d1b55fc6937d310a50317dcf312c64073ef0dba60"} Jul 17 22:23:00.877645 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:00.877621 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979024659776" Jul 17 22:23:00.896973 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:00.896942 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7321-139979024659776_d4265121-e1e6-4fa6-963b-2bd1b3cc432c/test-curl-7321-139979024659776/0.log" Jul 17 22:23:00.921749 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:00.921715 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7321-139979024659776"] Jul 17 22:23:00.924680 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:00.924650 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7321-139979024659776"] Jul 17 22:23:01.009506 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:01.009415 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q8pf\" (UniqueName: \"kubernetes.io/projected/d4265121-e1e6-4fa6-963b-2bd1b3cc432c-kube-api-access-9q8pf\") pod \"d4265121-e1e6-4fa6-963b-2bd1b3cc432c\" (UID: \"d4265121-e1e6-4fa6-963b-2bd1b3cc432c\") " Jul 17 22:23:01.011682 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:01.011658 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4265121-e1e6-4fa6-963b-2bd1b3cc432c-kube-api-access-9q8pf" (OuterVolumeSpecName: "kube-api-access-9q8pf") pod "d4265121-e1e6-4fa6-963b-2bd1b3cc432c" (UID: "d4265121-e1e6-4fa6-963b-2bd1b3cc432c"). InnerVolumeSpecName "kube-api-access-9q8pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:23:01.024086 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:01.024056 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4265121-e1e6-4fa6-963b-2bd1b3cc432c" path="/var/lib/kubelet/pods/d4265121-e1e6-4fa6-963b-2bd1b3cc432c/volumes" Jul 17 22:23:01.039145 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:01.039117 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7321-139979024659776"] Jul 17 22:23:01.039538 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:01.039524 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4265121-e1e6-4fa6-963b-2bd1b3cc432c" containerName="test-curl-7321-139979024659776" Jul 17 22:23:01.039587 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:01.039539 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4265121-e1e6-4fa6-963b-2bd1b3cc432c" containerName="test-curl-7321-139979024659776" Jul 17 22:23:01.039623 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:01.039609 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4265121-e1e6-4fa6-963b-2bd1b3cc432c" containerName="test-curl-7321-139979024659776" Jul 17 22:23:01.042609 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:01.042592 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979024659776" Jul 17 22:23:01.048945 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:01.048922 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7321-139979024659776"] Jul 17 22:23:01.110457 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:01.110384 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhdnh\" (UniqueName: \"kubernetes.io/projected/66f754e8-6e48-4ada-b302-e6cce1122f4f-kube-api-access-bhdnh\") pod \"test-curl-7321-139979024659776\" (UID: \"66f754e8-6e48-4ada-b302-e6cce1122f4f\") " pod="opendatahub/test-curl-7321-139979024659776" Jul 17 22:23:01.110911 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:01.110889 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q8pf\" (UniqueName: \"kubernetes.io/projected/d4265121-e1e6-4fa6-963b-2bd1b3cc432c-kube-api-access-9q8pf\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:23:01.211669 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:01.211631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhdnh\" (UniqueName: \"kubernetes.io/projected/66f754e8-6e48-4ada-b302-e6cce1122f4f-kube-api-access-bhdnh\") pod \"test-curl-7321-139979024659776\" (UID: \"66f754e8-6e48-4ada-b302-e6cce1122f4f\") " pod="opendatahub/test-curl-7321-139979024659776" Jul 17 22:23:01.219963 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:01.219926 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhdnh\" (UniqueName: \"kubernetes.io/projected/66f754e8-6e48-4ada-b302-e6cce1122f4f-kube-api-access-bhdnh\") pod \"test-curl-7321-139979024659776\" (UID: \"66f754e8-6e48-4ada-b302-e6cce1122f4f\") " pod="opendatahub/test-curl-7321-139979024659776" Jul 17 22:23:01.354773 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:01.354729 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979024659776" Jul 17 22:23:01.489961 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:01.489933 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7321-139979024659776"] Jul 17 22:23:01.491853 ip-10-0-130-56 kubenswrapper[2580]: W0717 22:23:01.491821 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66f754e8_6e48_4ada_b302_e6cce1122f4f.slice/crio-a76dc6b62528c736fe01b5507e17c191e060105dc39ad7adf3970ce8d47d0751 WatchSource:0}: Error finding container a76dc6b62528c736fe01b5507e17c191e060105dc39ad7adf3970ce8d47d0751: Status 404 returned error can't find the container with id a76dc6b62528c736fe01b5507e17c191e060105dc39ad7adf3970ce8d47d0751 Jul 17 22:23:01.742222 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:01.742136 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979024659776" Jul 17 22:23:01.742222 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:01.742164 2580 scope.go:117] "RemoveContainer" containerID="4bf55020d45e20635f826b8d1b55fc6937d310a50317dcf312c64073ef0dba60" Jul 17 22:23:01.743829 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:01.743652 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7321-139979024659776" event={"ID":"66f754e8-6e48-4ada-b302-e6cce1122f4f","Type":"ContainerStarted","Data":"a76dc6b62528c736fe01b5507e17c191e060105dc39ad7adf3970ce8d47d0751"} Jul 17 22:23:02.752026 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:02.751990 2580 generic.go:356] "Generic (PLEG): container finished" podID="66f754e8-6e48-4ada-b302-e6cce1122f4f" containerID="668ce11e325c1de326eb82b6c69ea15dd8e9b9d8d4c6a446fbf363d56ff4c168" exitCode=0 Jul 17 22:23:02.752456 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:02.752080 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7321-139979024659776" event={"ID":"66f754e8-6e48-4ada-b302-e6cce1122f4f","Type":"ContainerDied","Data":"668ce11e325c1de326eb82b6c69ea15dd8e9b9d8d4c6a446fbf363d56ff4c168"} Jul 17 22:23:03.893010 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:03.892983 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979024659776" Jul 17 22:23:03.912946 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:03.912916 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7321-139979024659776_66f754e8-6e48-4ada-b302-e6cce1122f4f/test-curl-7321-139979024659776/0.log" Jul 17 22:23:03.938832 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:03.938801 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7321-139979024659776"] Jul 17 22:23:03.941591 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:03.941565 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7321-139979024659776"] Jul 17 22:23:04.039042 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:04.039010 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhdnh\" (UniqueName: \"kubernetes.io/projected/66f754e8-6e48-4ada-b302-e6cce1122f4f-kube-api-access-bhdnh\") pod \"66f754e8-6e48-4ada-b302-e6cce1122f4f\" (UID: \"66f754e8-6e48-4ada-b302-e6cce1122f4f\") " Jul 17 22:23:04.041271 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:04.041247 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f754e8-6e48-4ada-b302-e6cce1122f4f-kube-api-access-bhdnh" (OuterVolumeSpecName: "kube-api-access-bhdnh") pod "66f754e8-6e48-4ada-b302-e6cce1122f4f" (UID: "66f754e8-6e48-4ada-b302-e6cce1122f4f"). InnerVolumeSpecName "kube-api-access-bhdnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:23:04.058877 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:04.058842 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7321-139979024660784"] Jul 17 22:23:04.059224 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:04.059211 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66f754e8-6e48-4ada-b302-e6cce1122f4f" containerName="test-curl-7321-139979024659776" Jul 17 22:23:04.059313 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:04.059226 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="66f754e8-6e48-4ada-b302-e6cce1122f4f" containerName="test-curl-7321-139979024659776" Jul 17 22:23:04.059350 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:04.059315 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="66f754e8-6e48-4ada-b302-e6cce1122f4f" containerName="test-curl-7321-139979024659776" Jul 17 22:23:04.062255 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:04.062237 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979024660784" Jul 17 22:23:04.068590 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:04.068564 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7321-139979024660784"] Jul 17 22:23:04.140572 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:04.140536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlzqw\" (UniqueName: \"kubernetes.io/projected/b39c820d-0ad3-4c55-a652-0c7c63a6c7d0-kube-api-access-vlzqw\") pod \"test-curl-7321-139979024660784\" (UID: \"b39c820d-0ad3-4c55-a652-0c7c63a6c7d0\") " pod="opendatahub/test-curl-7321-139979024660784" Jul 17 22:23:04.140753 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:04.140668 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhdnh\" (UniqueName: \"kubernetes.io/projected/66f754e8-6e48-4ada-b302-e6cce1122f4f-kube-api-access-bhdnh\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:23:04.241875 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:04.241841 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlzqw\" (UniqueName: \"kubernetes.io/projected/b39c820d-0ad3-4c55-a652-0c7c63a6c7d0-kube-api-access-vlzqw\") pod \"test-curl-7321-139979024660784\" (UID: \"b39c820d-0ad3-4c55-a652-0c7c63a6c7d0\") " pod="opendatahub/test-curl-7321-139979024660784" Jul 17 22:23:04.250759 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:04.250731 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlzqw\" (UniqueName: \"kubernetes.io/projected/b39c820d-0ad3-4c55-a652-0c7c63a6c7d0-kube-api-access-vlzqw\") pod \"test-curl-7321-139979024660784\" (UID: \"b39c820d-0ad3-4c55-a652-0c7c63a6c7d0\") " pod="opendatahub/test-curl-7321-139979024660784" Jul 17 22:23:04.373229 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:04.373141 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979024660784" Jul 17 22:23:04.512886 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:04.512859 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7321-139979024660784"] Jul 17 22:23:04.514499 ip-10-0-130-56 kubenswrapper[2580]: W0717 22:23:04.514475 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb39c820d_0ad3_4c55_a652_0c7c63a6c7d0.slice/crio-0067a5bc89f265df8ba173a3520488aa0e9c899780e43da132088327fda4dbd6 WatchSource:0}: Error finding container 0067a5bc89f265df8ba173a3520488aa0e9c899780e43da132088327fda4dbd6: Status 404 returned error can't find the container with id 0067a5bc89f265df8ba173a3520488aa0e9c899780e43da132088327fda4dbd6 Jul 17 22:23:04.764752 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:04.764727 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979024659776" Jul 17 22:23:04.764913 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:04.764731 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a76dc6b62528c736fe01b5507e17c191e060105dc39ad7adf3970ce8d47d0751" Jul 17 22:23:04.765945 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:04.765917 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7321-139979024660784" event={"ID":"b39c820d-0ad3-4c55-a652-0c7c63a6c7d0","Type":"ContainerStarted","Data":"0067a5bc89f265df8ba173a3520488aa0e9c899780e43da132088327fda4dbd6"} Jul 17 22:23:04.775251 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:23:04.775222 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7321-139979024659776\" is forbidden: User \"system:node:ip-10-0-130-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-56.ec2.internal' and this object" podUID="66f754e8-6e48-4ada-b302-e6cce1122f4f" pod="opendatahub/test-curl-7321-139979024659776" Jul 17 22:23:05.024602 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:05.024519 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66f754e8-6e48-4ada-b302-e6cce1122f4f" path="/var/lib/kubelet/pods/66f754e8-6e48-4ada-b302-e6cce1122f4f/volumes" Jul 17 22:23:05.773709 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:05.773677 2580 generic.go:356] "Generic (PLEG): container finished" podID="b39c820d-0ad3-4c55-a652-0c7c63a6c7d0" containerID="5bb0c840a343eba2d40d6e18edd44020274bd73d5fe1c590c84eed52115e6943" exitCode=0 Jul 17 22:23:05.773903 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:05.773761 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7321-139979024660784" event={"ID":"b39c820d-0ad3-4c55-a652-0c7c63a6c7d0","Type":"ContainerDied","Data":"5bb0c840a343eba2d40d6e18edd44020274bd73d5fe1c590c84eed52115e6943"} Jul 17 22:23:06.917445 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:06.917415 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979024660784" Jul 17 22:23:06.939133 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:06.939109 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7321-139979024660784_b39c820d-0ad3-4c55-a652-0c7c63a6c7d0/test-curl-7321-139979024660784/0.log" Jul 17 22:23:06.964485 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:06.964454 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7321-139979024660784"] Jul 17 22:23:06.967577 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:06.967551 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7321-139979024660784"] Jul 17 22:23:07.069149 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:07.069062 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlzqw\" (UniqueName: \"kubernetes.io/projected/b39c820d-0ad3-4c55-a652-0c7c63a6c7d0-kube-api-access-vlzqw\") pod \"b39c820d-0ad3-4c55-a652-0c7c63a6c7d0\" (UID: \"b39c820d-0ad3-4c55-a652-0c7c63a6c7d0\") " Jul 17 22:23:07.071385 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:07.071355 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39c820d-0ad3-4c55-a652-0c7c63a6c7d0-kube-api-access-vlzqw" (OuterVolumeSpecName: "kube-api-access-vlzqw") pod "b39c820d-0ad3-4c55-a652-0c7c63a6c7d0" (UID: "b39c820d-0ad3-4c55-a652-0c7c63a6c7d0"). InnerVolumeSpecName "kube-api-access-vlzqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:23:07.100261 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:07.100227 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7321-139979024660784"] Jul 17 22:23:07.100647 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:07.100633 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b39c820d-0ad3-4c55-a652-0c7c63a6c7d0" containerName="test-curl-7321-139979024660784" Jul 17 22:23:07.100702 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:07.100648 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b39c820d-0ad3-4c55-a652-0c7c63a6c7d0" containerName="test-curl-7321-139979024660784" Jul 17 22:23:07.100736 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:07.100723 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b39c820d-0ad3-4c55-a652-0c7c63a6c7d0" containerName="test-curl-7321-139979024660784" Jul 17 22:23:07.103773 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:07.103757 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979024660784" Jul 17 22:23:07.108230 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:07.108204 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7321-139979024660784"] Jul 17 22:23:07.169937 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:07.169908 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xfg9\" (UniqueName: \"kubernetes.io/projected/64f042a1-e3f6-4e54-982c-049ef49a1338-kube-api-access-7xfg9\") pod \"test-curl-7321-139979024660784\" (UID: \"64f042a1-e3f6-4e54-982c-049ef49a1338\") " pod="opendatahub/test-curl-7321-139979024660784" Jul 17 22:23:07.170131 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:07.169992 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlzqw\" (UniqueName: \"kubernetes.io/projected/b39c820d-0ad3-4c55-a652-0c7c63a6c7d0-kube-api-access-vlzqw\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:23:07.271516 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:07.271482 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xfg9\" (UniqueName: \"kubernetes.io/projected/64f042a1-e3f6-4e54-982c-049ef49a1338-kube-api-access-7xfg9\") pod \"test-curl-7321-139979024660784\" (UID: \"64f042a1-e3f6-4e54-982c-049ef49a1338\") " pod="opendatahub/test-curl-7321-139979024660784" Jul 17 22:23:07.279313 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:07.279288 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xfg9\" (UniqueName: \"kubernetes.io/projected/64f042a1-e3f6-4e54-982c-049ef49a1338-kube-api-access-7xfg9\") pod \"test-curl-7321-139979024660784\" (UID: \"64f042a1-e3f6-4e54-982c-049ef49a1338\") " pod="opendatahub/test-curl-7321-139979024660784" Jul 17 22:23:07.416683 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:07.416573 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979024660784" Jul 17 22:23:07.551557 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:07.551532 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7321-139979024660784"] Jul 17 22:23:07.553096 ip-10-0-130-56 kubenswrapper[2580]: W0717 22:23:07.553068 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64f042a1_e3f6_4e54_982c_049ef49a1338.slice/crio-574d760a74b0af8cb24169904e69ee341efd0b1de59d30aa362c5212ad900978 WatchSource:0}: Error finding container 574d760a74b0af8cb24169904e69ee341efd0b1de59d30aa362c5212ad900978: Status 404 returned error can't find the container with id 574d760a74b0af8cb24169904e69ee341efd0b1de59d30aa362c5212ad900978 Jul 17 22:23:07.784727 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:07.784689 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7321-139979024660784" event={"ID":"64f042a1-e3f6-4e54-982c-049ef49a1338","Type":"ContainerStarted","Data":"574d760a74b0af8cb24169904e69ee341efd0b1de59d30aa362c5212ad900978"} Jul 17 22:23:07.786061 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:07.786013 2580 scope.go:117] "RemoveContainer" containerID="5bb0c840a343eba2d40d6e18edd44020274bd73d5fe1c590c84eed52115e6943" Jul 17 22:23:07.786131 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:07.786055 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979024660784" Jul 17 22:23:08.795728 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:08.795690 2580 generic.go:356] "Generic (PLEG): container finished" podID="64f042a1-e3f6-4e54-982c-049ef49a1338" containerID="c07ae547c6f304a983850e4d5c66b3b8dd35c380ca556cf334045872a31dcddd" exitCode=0 Jul 17 22:23:08.796161 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:08.795777 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7321-139979024660784" event={"ID":"64f042a1-e3f6-4e54-982c-049ef49a1338","Type":"ContainerDied","Data":"c07ae547c6f304a983850e4d5c66b3b8dd35c380ca556cf334045872a31dcddd"} Jul 17 22:23:09.024103 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:09.024054 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b39c820d-0ad3-4c55-a652-0c7c63a6c7d0" path="/var/lib/kubelet/pods/b39c820d-0ad3-4c55-a652-0c7c63a6c7d0/volumes" Jul 17 22:23:09.939723 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:09.939697 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979024660784" Jul 17 22:23:09.963813 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:09.963779 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7321-139979024660784_64f042a1-e3f6-4e54-982c-049ef49a1338/test-curl-7321-139979024660784/0.log" Jul 17 22:23:09.990564 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:09.990528 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7321-139979024660784"] Jul 17 22:23:09.994731 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:09.994706 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7321-139979024660784"] Jul 17 22:23:10.100118 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:10.100020 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xfg9\" (UniqueName: \"kubernetes.io/projected/64f042a1-e3f6-4e54-982c-049ef49a1338-kube-api-access-7xfg9\") pod \"64f042a1-e3f6-4e54-982c-049ef49a1338\" (UID: \"64f042a1-e3f6-4e54-982c-049ef49a1338\") " Jul 17 22:23:10.102261 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:10.102228 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f042a1-e3f6-4e54-982c-049ef49a1338-kube-api-access-7xfg9" (OuterVolumeSpecName: "kube-api-access-7xfg9") pod "64f042a1-e3f6-4e54-982c-049ef49a1338" (UID: "64f042a1-e3f6-4e54-982c-049ef49a1338"). InnerVolumeSpecName "kube-api-access-7xfg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:23:10.201208 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:10.201174 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xfg9\" (UniqueName: \"kubernetes.io/projected/64f042a1-e3f6-4e54-982c-049ef49a1338-kube-api-access-7xfg9\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:23:10.655530 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:10.655498 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:23:10.755079 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:10.755047 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:23:10.809210 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:10.809181 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7321-139979024660784" Jul 17 22:23:10.809385 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:10.809185 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="574d760a74b0af8cb24169904e69ee341efd0b1de59d30aa362c5212ad900978" Jul 17 22:23:10.818797 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:23:10.818768 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7321-139979024660784\" is forbidden: User \"system:node:ip-10-0-130-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-56.ec2.internal' and this object" podUID="64f042a1-e3f6-4e54-982c-049ef49a1338" pod="opendatahub/test-curl-7321-139979024660784" Jul 17 22:23:11.024561 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:11.024533 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64f042a1-e3f6-4e54-982c-049ef49a1338" path="/var/lib/kubelet/pods/64f042a1-e3f6-4e54-982c-049ef49a1338/volumes" Jul 17 22:23:20.565672 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:20.565635 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5"] Jul 17 22:23:20.566191 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:20.565914 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" podUID="3d05686e-37e8-4b0d-bc5c-9a7d6e97399d" containerName="maas-api" containerID="cri-o://dbd916a94bf366ca9decfb262e8cc3e77f6362443721407c61e7000e084bb955" gracePeriod=30 Jul 17 22:23:20.600882 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:20.600841 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:23:20.820867 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:20.820799 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" Jul 17 22:23:20.864058 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:20.864030 2580 generic.go:356] "Generic (PLEG): container finished" podID="3d05686e-37e8-4b0d-bc5c-9a7d6e97399d" containerID="dbd916a94bf366ca9decfb262e8cc3e77f6362443721407c61e7000e084bb955" exitCode=0 Jul 17 22:23:20.864203 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:20.864079 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" Jul 17 22:23:20.864203 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:20.864127 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" event={"ID":"3d05686e-37e8-4b0d-bc5c-9a7d6e97399d","Type":"ContainerDied","Data":"dbd916a94bf366ca9decfb262e8cc3e77f6362443721407c61e7000e084bb955"} Jul 17 22:23:20.864203 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:20.864184 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5" event={"ID":"3d05686e-37e8-4b0d-bc5c-9a7d6e97399d","Type":"ContainerDied","Data":"68a3eb7c5a2253d1405ae1da7586893b6ca54d1197b2f98766d0e0dc094d4152"} Jul 17 22:23:20.864325 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:20.864214 2580 scope.go:117] "RemoveContainer" containerID="dbd916a94bf366ca9decfb262e8cc3e77f6362443721407c61e7000e084bb955" Jul 17 22:23:20.872954 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:20.872934 2580 scope.go:117] "RemoveContainer" containerID="dbd916a94bf366ca9decfb262e8cc3e77f6362443721407c61e7000e084bb955" Jul 17 22:23:20.873258 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:23:20.873235 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd916a94bf366ca9decfb262e8cc3e77f6362443721407c61e7000e084bb955\": container with ID starting with dbd916a94bf366ca9decfb262e8cc3e77f6362443721407c61e7000e084bb955 not found: ID does not exist" containerID="dbd916a94bf366ca9decfb262e8cc3e77f6362443721407c61e7000e084bb955" Jul 17 22:23:20.873301 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:20.873268 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd916a94bf366ca9decfb262e8cc3e77f6362443721407c61e7000e084bb955"} err="failed to get container status \"dbd916a94bf366ca9decfb262e8cc3e77f6362443721407c61e7000e084bb955\": rpc error: code = NotFound desc = could not find container \"dbd916a94bf366ca9decfb262e8cc3e77f6362443721407c61e7000e084bb955\": container with ID starting with dbd916a94bf366ca9decfb262e8cc3e77f6362443721407c61e7000e084bb955 not found: ID does not exist" Jul 17 22:23:21.002292 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:21.002253 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phrbb\" (UniqueName: \"kubernetes.io/projected/3d05686e-37e8-4b0d-bc5c-9a7d6e97399d-kube-api-access-phrbb\") pod \"3d05686e-37e8-4b0d-bc5c-9a7d6e97399d\" (UID: \"3d05686e-37e8-4b0d-bc5c-9a7d6e97399d\") " Jul 17 22:23:21.002292 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:21.002296 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d05686e-37e8-4b0d-bc5c-9a7d6e97399d-openshift-service-ca\") pod \"3d05686e-37e8-4b0d-bc5c-9a7d6e97399d\" (UID: \"3d05686e-37e8-4b0d-bc5c-9a7d6e97399d\") " Jul 17 22:23:21.002518 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:21.002413 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d05686e-37e8-4b0d-bc5c-9a7d6e97399d-maas-api-tls\") pod \"3d05686e-37e8-4b0d-bc5c-9a7d6e97399d\" (UID: \"3d05686e-37e8-4b0d-bc5c-9a7d6e97399d\") " Jul 17 22:23:21.002744 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:21.002708 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d05686e-37e8-4b0d-bc5c-9a7d6e97399d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3d05686e-37e8-4b0d-bc5c-9a7d6e97399d" (UID: "3d05686e-37e8-4b0d-bc5c-9a7d6e97399d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:23:21.004684 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:21.004657 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d05686e-37e8-4b0d-bc5c-9a7d6e97399d-kube-api-access-phrbb" (OuterVolumeSpecName: "kube-api-access-phrbb") pod "3d05686e-37e8-4b0d-bc5c-9a7d6e97399d" (UID: "3d05686e-37e8-4b0d-bc5c-9a7d6e97399d"). InnerVolumeSpecName "kube-api-access-phrbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:23:21.004807 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:21.004690 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d05686e-37e8-4b0d-bc5c-9a7d6e97399d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3d05686e-37e8-4b0d-bc5c-9a7d6e97399d" (UID: "3d05686e-37e8-4b0d-bc5c-9a7d6e97399d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:23:21.103364 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:21.103274 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phrbb\" (UniqueName: \"kubernetes.io/projected/3d05686e-37e8-4b0d-bc5c-9a7d6e97399d-kube-api-access-phrbb\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:23:21.103364 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:21.103306 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d05686e-37e8-4b0d-bc5c-9a7d6e97399d-openshift-service-ca\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:23:21.103364 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:21.103319 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d05686e-37e8-4b0d-bc5c-9a7d6e97399d-maas-api-tls\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:23:21.179817 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:21.179786 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5"] Jul 17 22:23:21.183412 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:21.183369 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8e6a3a-5f8b7979d-rfsd5"] Jul 17 22:23:23.024580 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:23.024538 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d05686e-37e8-4b0d-bc5c-9a7d6e97399d" path="/var/lib/kubelet/pods/3d05686e-37e8-4b0d-bc5c-9a7d6e97399d/volumes" Jul 17 22:23:25.295006 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:25.294973 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:23:31.160757 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:31.160723 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:23:31.260911 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:31.260869 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:23:35.997969 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:35.997921 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:23:36.240540 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:36.240496 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:23:37.009772 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:37.009738 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:23:37.111334 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:37.111300 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:23:46.757163 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:46.757127 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m"] Jul 17 22:23:46.757742 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:46.757443 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" podUID="713c23d7-0f21-4eea-b69e-71d6ea4c476f" containerName="maas-api" containerID="cri-o://6c94a606a6ba3eda8fb5ed10e5607648f609b1618578a33679e15649ba329f44" gracePeriod=30 Jul 17 22:23:46.786033 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:46.785989 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:23:47.208405 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:47.208362 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" Jul 17 22:23:47.345366 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:47.345281 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj5c2\" (UniqueName: \"kubernetes.io/projected/713c23d7-0f21-4eea-b69e-71d6ea4c476f-kube-api-access-lj5c2\") pod \"713c23d7-0f21-4eea-b69e-71d6ea4c476f\" (UID: \"713c23d7-0f21-4eea-b69e-71d6ea4c476f\") " Jul 17 22:23:47.345366 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:47.345361 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/713c23d7-0f21-4eea-b69e-71d6ea4c476f-maas-api-tls\") pod \"713c23d7-0f21-4eea-b69e-71d6ea4c476f\" (UID: \"713c23d7-0f21-4eea-b69e-71d6ea4c476f\") " Jul 17 22:23:47.345654 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:47.345443 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/713c23d7-0f21-4eea-b69e-71d6ea4c476f-openshift-service-ca\") pod \"713c23d7-0f21-4eea-b69e-71d6ea4c476f\" (UID: \"713c23d7-0f21-4eea-b69e-71d6ea4c476f\") " Jul 17 22:23:47.345821 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:47.345796 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/713c23d7-0f21-4eea-b69e-71d6ea4c476f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "713c23d7-0f21-4eea-b69e-71d6ea4c476f" (UID: "713c23d7-0f21-4eea-b69e-71d6ea4c476f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:23:47.347640 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:47.347609 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/713c23d7-0f21-4eea-b69e-71d6ea4c476f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "713c23d7-0f21-4eea-b69e-71d6ea4c476f" (UID: "713c23d7-0f21-4eea-b69e-71d6ea4c476f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:23:47.347750 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:47.347730 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713c23d7-0f21-4eea-b69e-71d6ea4c476f-kube-api-access-lj5c2" (OuterVolumeSpecName: "kube-api-access-lj5c2") pod "713c23d7-0f21-4eea-b69e-71d6ea4c476f" (UID: "713c23d7-0f21-4eea-b69e-71d6ea4c476f"). InnerVolumeSpecName "kube-api-access-lj5c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:23:47.446732 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:47.446695 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj5c2\" (UniqueName: \"kubernetes.io/projected/713c23d7-0f21-4eea-b69e-71d6ea4c476f-kube-api-access-lj5c2\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:23:47.446732 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:47.446725 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/713c23d7-0f21-4eea-b69e-71d6ea4c476f-maas-api-tls\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:23:47.446732 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:47.446734 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/713c23d7-0f21-4eea-b69e-71d6ea4c476f-openshift-service-ca\") on node \"ip-10-0-130-56.ec2.internal\" DevicePath \"\"" Jul 17 22:23:48.022078 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:48.022045 2580 generic.go:356] "Generic (PLEG): container finished" podID="713c23d7-0f21-4eea-b69e-71d6ea4c476f" containerID="6c94a606a6ba3eda8fb5ed10e5607648f609b1618578a33679e15649ba329f44" exitCode=0 Jul 17 22:23:48.022473 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:48.022091 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" Jul 17 22:23:48.022473 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:48.022119 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" event={"ID":"713c23d7-0f21-4eea-b69e-71d6ea4c476f","Type":"ContainerDied","Data":"6c94a606a6ba3eda8fb5ed10e5607648f609b1618578a33679e15649ba329f44"} Jul 17 22:23:48.022473 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:48.022164 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m" event={"ID":"713c23d7-0f21-4eea-b69e-71d6ea4c476f","Type":"ContainerDied","Data":"efe7b9919d122d1efd34db131f53925b7e24b161ff5fd7c5dca5cfdb4838d0be"} Jul 17 22:23:48.022473 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:48.022186 2580 scope.go:117] "RemoveContainer" containerID="6c94a606a6ba3eda8fb5ed10e5607648f609b1618578a33679e15649ba329f44" Jul 17 22:23:48.031221 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:48.031203 2580 scope.go:117] "RemoveContainer" containerID="6c94a606a6ba3eda8fb5ed10e5607648f609b1618578a33679e15649ba329f44" Jul 17 22:23:48.031513 ip-10-0-130-56 kubenswrapper[2580]: E0717 22:23:48.031488 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c94a606a6ba3eda8fb5ed10e5607648f609b1618578a33679e15649ba329f44\": container with ID starting with 6c94a606a6ba3eda8fb5ed10e5607648f609b1618578a33679e15649ba329f44 not found: ID does not exist" containerID="6c94a606a6ba3eda8fb5ed10e5607648f609b1618578a33679e15649ba329f44" Jul 17 22:23:48.031596 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:48.031548 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c94a606a6ba3eda8fb5ed10e5607648f609b1618578a33679e15649ba329f44"} err="failed to get container status \"6c94a606a6ba3eda8fb5ed10e5607648f609b1618578a33679e15649ba329f44\": rpc error: code = NotFound desc = could not find container \"6c94a606a6ba3eda8fb5ed10e5607648f609b1618578a33679e15649ba329f44\": container with ID starting with 6c94a606a6ba3eda8fb5ed10e5607648f609b1618578a33679e15649ba329f44 not found: ID does not exist" Jul 17 22:23:48.043241 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:48.043210 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m"] Jul 17 22:23:48.047227 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:48.047203 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5401e5-bbc57c7c5-z6h8m"] Jul 17 22:23:49.024227 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:49.024192 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="713c23d7-0f21-4eea-b69e-71d6ea4c476f" path="/var/lib/kubelet/pods/713c23d7-0f21-4eea-b69e-71d6ea4c476f/volumes" Jul 17 22:23:51.457950 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:51.457914 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:23:57.235940 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:57.235903 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:23:57.336645 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:23:57.336602 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:24:02.554547 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:02.554498 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:24:03.178518 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:03.178461 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:24:03.648656 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:03.648626 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-689b579cb6-vzbzb_5da48d3c-c7f0-4a60-96fd-361218f717bb/authorino/0.log" Jul 17 22:24:09.224323 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:09.224292 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86d8757495-sqn2m_0f885ea0-1e53-4a18-975d-be0ea8fe68ed/manager/0.log" Jul 17 22:24:09.338713 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:09.338685 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-jwq2n_3f2fc54c-9516-4d2c-b5ac-0910dca90e4b/server/0.log" Jul 17 22:24:09.464310 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:09.464277 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-54bpn_88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb/manager/0.log" Jul 17 22:24:09.707422 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:09.707359 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-964b768d-r6hsf_48e2e9b0-f33c-418d-a7ce-852d2b31d50a/copy-manifests/0.log" Jul 17 22:24:09.721116 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:09.721089 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-964b768d-r6hsf_48e2e9b0-f33c-418d-a7ce-852d2b31d50a/manager/0.log" Jul 17 22:24:10.218124 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:10.218091 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7d47bfd4-vbd66_636cf20d-4276-45e6-87ec-c8d24c30724b/maas-api/0.log" Jul 17 22:24:12.427785 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:12.427759 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj_2b53e33e-b212-4d0e-a4f5-af90b6f854b1/util/0.log" Jul 17 22:24:12.434494 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:12.434462 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj_2b53e33e-b212-4d0e-a4f5-af90b6f854b1/pull/0.log" Jul 17 22:24:12.440607 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:12.440590 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj_2b53e33e-b212-4d0e-a4f5-af90b6f854b1/extract/0.log" Jul 17 22:24:12.549383 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:12.549345 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z_deb01aec-306e-4a20-bee6-172990a40348/util/0.log" Jul 17 22:24:12.555480 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:12.555445 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z_deb01aec-306e-4a20-bee6-172990a40348/pull/0.log" Jul 17 22:24:12.561851 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:12.561833 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z_deb01aec-306e-4a20-bee6-172990a40348/extract/0.log" Jul 17 22:24:12.674503 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:12.674471 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl_6b06a62b-c6dd-4204-9984-180bb2320534/util/0.log" Jul 17 22:24:12.681020 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:12.680926 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl_6b06a62b-c6dd-4204-9984-180bb2320534/pull/0.log" Jul 17 22:24:12.687181 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:12.687157 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl_6b06a62b-c6dd-4204-9984-180bb2320534/extract/0.log" Jul 17 22:24:12.794203 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:12.794174 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9_3c5048e7-d826-4922-b3d5-29129e8cc5fd/util/0.log" Jul 17 22:24:12.800629 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:12.800609 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9_3c5048e7-d826-4922-b3d5-29129e8cc5fd/pull/0.log" Jul 17 22:24:12.806771 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:12.806751 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9_3c5048e7-d826-4922-b3d5-29129e8cc5fd/extract/0.log" Jul 17 22:24:12.922244 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:12.922218 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-689b579cb6-vzbzb_5da48d3c-c7f0-4a60-96fd-361218f717bb/authorino/0.log" Jul 17 22:24:13.260297 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:13.260264 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-svhjt_c58d0a39-8de8-460b-83fb-6474831ee5b0/kuadrant-console-plugin/0.log" Jul 17 22:24:14.413768 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:14.413736 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-788f6fc9d-h598z_f8f4fcdb-b37c-471e-89f6-850451fa8af4/kube-auth-proxy/0.log" Jul 17 22:24:15.366204 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:15.366167 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-fmzfl_7fb2954c-6a9a-498f-857a-18259d4402b3/main/0.log" Jul 17 22:24:15.597334 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:15.597307 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmq789_0e142f0b-c204-4279-ae86-259f295dfd78/main/0.log" Jul 17 22:24:15.817113 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:15.817081 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-9gp66_59c1a244-a0d1-4e65-9c4e-919ff3da1fd4/main/0.log" Jul 17 22:24:21.751640 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.751601 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5qg69/must-gather-pkc4q"] Jul 17 22:24:21.752033 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.751966 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d05686e-37e8-4b0d-bc5c-9a7d6e97399d" containerName="maas-api" Jul 17 22:24:21.752033 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.751977 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d05686e-37e8-4b0d-bc5c-9a7d6e97399d" containerName="maas-api" Jul 17 22:24:21.752033 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.751992 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f042a1-e3f6-4e54-982c-049ef49a1338" containerName="test-curl-7321-139979024660784" Jul 17 22:24:21.752033 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.752000 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f042a1-e3f6-4e54-982c-049ef49a1338" containerName="test-curl-7321-139979024660784" Jul 17 22:24:21.752033 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.752009 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="713c23d7-0f21-4eea-b69e-71d6ea4c476f" containerName="maas-api" Jul 17 22:24:21.752033 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.752015 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="713c23d7-0f21-4eea-b69e-71d6ea4c476f" containerName="maas-api" Jul 17 22:24:21.752226 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.752071 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="64f042a1-e3f6-4e54-982c-049ef49a1338" containerName="test-curl-7321-139979024660784" Jul 17 22:24:21.752226 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.752079 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d05686e-37e8-4b0d-bc5c-9a7d6e97399d" containerName="maas-api" Jul 17 22:24:21.752226 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.752086 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="713c23d7-0f21-4eea-b69e-71d6ea4c476f" containerName="maas-api" Jul 17 22:24:21.755121 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.755101 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5qg69/must-gather-pkc4q" Jul 17 22:24:21.757708 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.757683 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5qg69\"/\"default-dockercfg-ljcb8\"" Jul 17 22:24:21.757866 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.757769 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5qg69\"/\"openshift-service-ca.crt\"" Jul 17 22:24:21.757866 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.757799 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5qg69\"/\"kube-root-ca.crt\"" Jul 17 22:24:21.767923 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.767886 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5qg69/must-gather-pkc4q"] Jul 17 22:24:21.778234 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.778208 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:24:21.843831 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.843792 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" limit=1024 Jul 17 22:24:21.852891 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.852871 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl2fl\" (UniqueName: \"kubernetes.io/projected/014db589-a9ae-418f-8466-7f5b4ac53f27-kube-api-access-bl2fl\") pod \"must-gather-pkc4q\" (UID: \"014db589-a9ae-418f-8466-7f5b4ac53f27\") " pod="openshift-must-gather-5qg69/must-gather-pkc4q" Jul 17 22:24:21.852962 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.852918 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/014db589-a9ae-418f-8466-7f5b4ac53f27-must-gather-output\") pod \"must-gather-pkc4q\" (UID: \"014db589-a9ae-418f-8466-7f5b4ac53f27\") " pod="openshift-must-gather-5qg69/must-gather-pkc4q" Jul 17 22:24:21.954371 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.954327 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl2fl\" (UniqueName: \"kubernetes.io/projected/014db589-a9ae-418f-8466-7f5b4ac53f27-kube-api-access-bl2fl\") pod \"must-gather-pkc4q\" (UID: \"014db589-a9ae-418f-8466-7f5b4ac53f27\") " pod="openshift-must-gather-5qg69/must-gather-pkc4q" Jul 17 22:24:21.954371 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.954400 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/014db589-a9ae-418f-8466-7f5b4ac53f27-must-gather-output\") pod \"must-gather-pkc4q\" (UID: \"014db589-a9ae-418f-8466-7f5b4ac53f27\") " pod="openshift-must-gather-5qg69/must-gather-pkc4q" Jul 17 22:24:21.954737 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.954722 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/014db589-a9ae-418f-8466-7f5b4ac53f27-must-gather-output\") pod \"must-gather-pkc4q\" (UID: \"014db589-a9ae-418f-8466-7f5b4ac53f27\") " pod="openshift-must-gather-5qg69/must-gather-pkc4q" Jul 17 22:24:21.962834 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:21.962798 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl2fl\" (UniqueName: \"kubernetes.io/projected/014db589-a9ae-418f-8466-7f5b4ac53f27-kube-api-access-bl2fl\") pod \"must-gather-pkc4q\" (UID: \"014db589-a9ae-418f-8466-7f5b4ac53f27\") " pod="openshift-must-gather-5qg69/must-gather-pkc4q" Jul 17 22:24:22.069194 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:22.069154 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5qg69/must-gather-pkc4q" Jul 17 22:24:22.206651 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:22.206619 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5qg69/must-gather-pkc4q"] Jul 17 22:24:22.210241 ip-10-0-130-56 kubenswrapper[2580]: W0717 22:24:22.210211 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod014db589_a9ae_418f_8466_7f5b4ac53f27.slice/crio-cad054504250240cb7a54213d925e29b71e36aedbd59be1f2fc3eb65de3ef0db WatchSource:0}: Error finding container cad054504250240cb7a54213d925e29b71e36aedbd59be1f2fc3eb65de3ef0db: Status 404 returned error can't find the container with id cad054504250240cb7a54213d925e29b71e36aedbd59be1f2fc3eb65de3ef0db Jul 17 22:24:23.217184 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:23.217155 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5qg69/must-gather-pkc4q" event={"ID":"014db589-a9ae-418f-8466-7f5b4ac53f27","Type":"ContainerStarted","Data":"cad054504250240cb7a54213d925e29b71e36aedbd59be1f2fc3eb65de3ef0db"} Jul 17 22:24:24.227621 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:24.227563 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5qg69/must-gather-pkc4q" event={"ID":"014db589-a9ae-418f-8466-7f5b4ac53f27","Type":"ContainerStarted","Data":"813fd60568b774cf16eac5262d9a409112cb316f1b12d1f1d0bdeaca2ac55fcd"} Jul 17 22:24:24.227621 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:24.227626 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5qg69/must-gather-pkc4q" event={"ID":"014db589-a9ae-418f-8466-7f5b4ac53f27","Type":"ContainerStarted","Data":"e75951fb1f4b26403854f479736248b225e58a6080cb6e73c1ce9ec59182b44a"} Jul 17 22:24:24.245796 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:24.245734 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5qg69/must-gather-pkc4q" podStartSLOduration=2.316139335 podStartE2EDuration="3.245714116s" totalImagesPullingTime="929.574781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:24:24.243250094 +0000 UTC m=+3405.664969327" watchObservedRunningTime="2026-07-17 22:24:24.245714116 +0000 UTC m=+3405.667433350" Jul 17 22:24:24.678975 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:24.678934 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fl9qj_f44bc251-268b-42f5-b9d3-a742151884f9/global-pull-secret-syncer/0.log" Jul 17 22:24:24.763845 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:24.763806 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bvspz_cc3959d9-8af3-4c63-a611-9fb0321a7e47/konnectivity-agent/0.log" Jul 17 22:24:24.860050 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:24.860017 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-56.ec2.internal_dc49aa9b1d03314cbcca016a92d97157/haproxy/0.log" Jul 17 22:24:29.348047 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:29.347871 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj_2b53e33e-b212-4d0e-a4f5-af90b6f854b1/extract/0.log" Jul 17 22:24:29.368159 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:29.368006 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj_2b53e33e-b212-4d0e-a4f5-af90b6f854b1/util/0.log" Jul 17 22:24:29.388863 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:29.388834 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gxtgj_2b53e33e-b212-4d0e-a4f5-af90b6f854b1/pull/0.log" Jul 17 22:24:29.426739 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:29.426643 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z_deb01aec-306e-4a20-bee6-172990a40348/extract/0.log" Jul 17 22:24:29.453666 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:29.453626 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z_deb01aec-306e-4a20-bee6-172990a40348/util/0.log" Jul 17 22:24:29.482692 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:29.482656 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sr96z_deb01aec-306e-4a20-bee6-172990a40348/pull/0.log" Jul 17 22:24:29.532049 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:29.532016 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl_6b06a62b-c6dd-4204-9984-180bb2320534/extract/0.log" Jul 17 22:24:29.553224 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:29.553200 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl_6b06a62b-c6dd-4204-9984-180bb2320534/util/0.log" Jul 17 22:24:29.574650 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:29.574619 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734b6kl_6b06a62b-c6dd-4204-9984-180bb2320534/pull/0.log" Jul 17 22:24:29.600917 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:29.600827 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9_3c5048e7-d826-4922-b3d5-29129e8cc5fd/extract/0.log" Jul 17 22:24:29.627770 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:29.627739 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9_3c5048e7-d826-4922-b3d5-29129e8cc5fd/util/0.log" Jul 17 22:24:29.649940 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:29.649888 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17cmk9_3c5048e7-d826-4922-b3d5-29129e8cc5fd/pull/0.log" Jul 17 22:24:30.118414 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:30.117087 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-689b579cb6-vzbzb_5da48d3c-c7f0-4a60-96fd-361218f717bb/authorino/0.log" Jul 17 22:24:30.208540 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:30.208511 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-svhjt_c58d0a39-8de8-460b-83fb-6474831ee5b0/kuadrant-console-plugin/0.log" Jul 17 22:24:32.033445 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:32.033418 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aa077b26-a698-4b72-8f63-b8d3edf4d8ee/alertmanager/0.log" Jul 17 22:24:32.058706 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:32.058677 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aa077b26-a698-4b72-8f63-b8d3edf4d8ee/config-reloader/0.log" Jul 17 22:24:32.079845 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:32.079775 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aa077b26-a698-4b72-8f63-b8d3edf4d8ee/kube-rbac-proxy-web/0.log" Jul 17 22:24:32.103470 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:32.103404 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aa077b26-a698-4b72-8f63-b8d3edf4d8ee/kube-rbac-proxy/0.log" Jul 17 22:24:32.135228 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:32.135204 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aa077b26-a698-4b72-8f63-b8d3edf4d8ee/kube-rbac-proxy-metric/0.log" Jul 17 22:24:32.156195 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:32.156164 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aa077b26-a698-4b72-8f63-b8d3edf4d8ee/prom-label-proxy/0.log" Jul 17 22:24:32.184098 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:32.183989 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_aa077b26-a698-4b72-8f63-b8d3edf4d8ee/init-config-reloader/0.log" Jul 17 22:24:32.572212 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:32.572165 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zz8sh_31b165ae-1d35-4dfa-8ff9-f693f65bc628/node-exporter/0.log" Jul 17 22:24:32.594431 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:32.594277 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zz8sh_31b165ae-1d35-4dfa-8ff9-f693f65bc628/kube-rbac-proxy/0.log" Jul 17 22:24:32.619653 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:32.619626 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zz8sh_31b165ae-1d35-4dfa-8ff9-f693f65bc628/init-textfile/0.log" Jul 17 22:24:32.663854 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:32.663822 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-w8z5k_513ad778-cfa6-4e4f-8a6f-e8c0a08492f4/kube-rbac-proxy-main/0.log" Jul 17 22:24:32.687199 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:32.687164 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-w8z5k_513ad778-cfa6-4e4f-8a6f-e8c0a08492f4/kube-rbac-proxy-self/0.log" Jul 17 22:24:32.708871 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:32.708839 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-w8z5k_513ad778-cfa6-4e4f-8a6f-e8c0a08492f4/openshift-state-metrics/0.log" Jul 17 22:24:32.995145 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:32.995066 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6798878cfc-9kgql_73c52f87-4e3e-4427-8112-f1ca2c2fba25/telemeter-client/0.log" Jul 17 22:24:33.023296 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:33.023262 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6798878cfc-9kgql_73c52f87-4e3e-4427-8112-f1ca2c2fba25/reload/0.log" Jul 17 22:24:33.048212 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:33.048181 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6798878cfc-9kgql_73c52f87-4e3e-4427-8112-f1ca2c2fba25/kube-rbac-proxy/0.log" Jul 17 22:24:33.101039 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:33.101005 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76bb7cbc89-5flq2_8785431a-f6f7-4fc6-a97f-268e35ad79e9/thanos-query/0.log" Jul 17 22:24:33.122723 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:33.122690 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76bb7cbc89-5flq2_8785431a-f6f7-4fc6-a97f-268e35ad79e9/kube-rbac-proxy-web/0.log" Jul 17 22:24:33.146367 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:33.146336 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76bb7cbc89-5flq2_8785431a-f6f7-4fc6-a97f-268e35ad79e9/kube-rbac-proxy/0.log" Jul 17 22:24:33.170132 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:33.170095 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76bb7cbc89-5flq2_8785431a-f6f7-4fc6-a97f-268e35ad79e9/prom-label-proxy/0.log" Jul 17 22:24:33.196434 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:33.196351 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76bb7cbc89-5flq2_8785431a-f6f7-4fc6-a97f-268e35ad79e9/kube-rbac-proxy-rules/0.log" Jul 17 22:24:33.218306 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:33.218282 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76bb7cbc89-5flq2_8785431a-f6f7-4fc6-a97f-268e35ad79e9/kube-rbac-proxy-metrics/0.log" Jul 17 22:24:35.007607 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.007567 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6"] Jul 17 22:24:35.013165 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.013138 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" Jul 17 22:24:35.029202 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.029091 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6"] Jul 17 22:24:35.098295 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.098254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07983003-0068-4b4b-86ff-032f5b1b3316-sys\") pod \"perf-node-gather-daemonset-8cbj6\" (UID: \"07983003-0068-4b4b-86ff-032f5b1b3316\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" Jul 17 22:24:35.098658 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.098636 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07983003-0068-4b4b-86ff-032f5b1b3316-podres\") pod \"perf-node-gather-daemonset-8cbj6\" (UID: \"07983003-0068-4b4b-86ff-032f5b1b3316\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" Jul 17 22:24:35.098905 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.098879 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07983003-0068-4b4b-86ff-032f5b1b3316-proc\") pod \"perf-node-gather-daemonset-8cbj6\" (UID: \"07983003-0068-4b4b-86ff-032f5b1b3316\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" Jul 17 22:24:35.099022 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.098915 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92s9n\" (UniqueName: \"kubernetes.io/projected/07983003-0068-4b4b-86ff-032f5b1b3316-kube-api-access-92s9n\") pod \"perf-node-gather-daemonset-8cbj6\" (UID: \"07983003-0068-4b4b-86ff-032f5b1b3316\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" Jul 17 22:24:35.099022 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.098951 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07983003-0068-4b4b-86ff-032f5b1b3316-lib-modules\") pod \"perf-node-gather-daemonset-8cbj6\" (UID: \"07983003-0068-4b4b-86ff-032f5b1b3316\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" Jul 17 22:24:35.199764 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.199713 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07983003-0068-4b4b-86ff-032f5b1b3316-podres\") pod \"perf-node-gather-daemonset-8cbj6\" (UID: \"07983003-0068-4b4b-86ff-032f5b1b3316\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" Jul 17 22:24:35.199971 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.199783 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07983003-0068-4b4b-86ff-032f5b1b3316-proc\") pod \"perf-node-gather-daemonset-8cbj6\" (UID: \"07983003-0068-4b4b-86ff-032f5b1b3316\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" Jul 17 22:24:35.199971 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.199808 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92s9n\" (UniqueName: \"kubernetes.io/projected/07983003-0068-4b4b-86ff-032f5b1b3316-kube-api-access-92s9n\") pod \"perf-node-gather-daemonset-8cbj6\" (UID: \"07983003-0068-4b4b-86ff-032f5b1b3316\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" Jul 17 22:24:35.199971 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.199834 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07983003-0068-4b4b-86ff-032f5b1b3316-lib-modules\") pod \"perf-node-gather-daemonset-8cbj6\" (UID: \"07983003-0068-4b4b-86ff-032f5b1b3316\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" Jul 17 22:24:35.199971 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.199898 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07983003-0068-4b4b-86ff-032f5b1b3316-sys\") pod \"perf-node-gather-daemonset-8cbj6\" (UID: \"07983003-0068-4b4b-86ff-032f5b1b3316\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" Jul 17 22:24:35.199971 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.199904 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07983003-0068-4b4b-86ff-032f5b1b3316-podres\") pod \"perf-node-gather-daemonset-8cbj6\" (UID: \"07983003-0068-4b4b-86ff-032f5b1b3316\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" Jul 17 22:24:35.199971 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.199914 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07983003-0068-4b4b-86ff-032f5b1b3316-proc\") pod \"perf-node-gather-daemonset-8cbj6\" (UID: \"07983003-0068-4b4b-86ff-032f5b1b3316\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" Jul 17 22:24:35.200406 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.200051 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07983003-0068-4b4b-86ff-032f5b1b3316-sys\") pod \"perf-node-gather-daemonset-8cbj6\" (UID: \"07983003-0068-4b4b-86ff-032f5b1b3316\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" Jul 17 22:24:35.200406 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.200061 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07983003-0068-4b4b-86ff-032f5b1b3316-lib-modules\") pod \"perf-node-gather-daemonset-8cbj6\" (UID: \"07983003-0068-4b4b-86ff-032f5b1b3316\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" Jul 17 22:24:35.208523 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.208490 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92s9n\" (UniqueName: \"kubernetes.io/projected/07983003-0068-4b4b-86ff-032f5b1b3316-kube-api-access-92s9n\") pod \"perf-node-gather-daemonset-8cbj6\" (UID: \"07983003-0068-4b4b-86ff-032f5b1b3316\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" Jul 17 22:24:35.335434 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.335370 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" Jul 17 22:24:35.527431 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:35.527132 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6"] Jul 17 22:24:35.532428 ip-10-0-130-56 kubenswrapper[2580]: W0717 22:24:35.532140 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod07983003_0068_4b4b_86ff_032f5b1b3316.slice/crio-070aeb4b33e829a07a23b1bd7206c843744fbc17a10084092024090655085fe0 WatchSource:0}: Error finding container 070aeb4b33e829a07a23b1bd7206c843744fbc17a10084092024090655085fe0: Status 404 returned error can't find the container with id 070aeb4b33e829a07a23b1bd7206c843744fbc17a10084092024090655085fe0 Jul 17 22:24:36.328934 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:36.328885 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" event={"ID":"07983003-0068-4b4b-86ff-032f5b1b3316","Type":"ContainerStarted","Data":"74d88e3b558916571dd793b9dfbf1f1d711874f26af6ef1b21c1100375c458eb"} Jul 17 22:24:36.329480 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:36.328964 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" event={"ID":"07983003-0068-4b4b-86ff-032f5b1b3316","Type":"ContainerStarted","Data":"070aeb4b33e829a07a23b1bd7206c843744fbc17a10084092024090655085fe0"} Jul 17 22:24:36.329480 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:36.329212 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" Jul 17 22:24:36.347972 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:36.347917 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" podStartSLOduration=2.347895763 podStartE2EDuration="2.347895763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 22:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:24:36.345770334 +0000 UTC m=+3417.767489568" watchObservedRunningTime="2026-07-17 22:24:36.347895763 +0000 UTC m=+3417.769614996" Jul 17 22:24:37.381354 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:37.381313 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ztjvd_b229eabe-6732-4bdf-b6c8-57555f105efa/dns/0.log" Jul 17 22:24:37.403073 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:37.403025 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ztjvd_b229eabe-6732-4bdf-b6c8-57555f105efa/kube-rbac-proxy/0.log" Jul 17 22:24:37.425208 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:37.425178 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4fgf5_3e6f72d2-a524-4587-a7f5-278128245a09/dns-node-resolver/0.log" Jul 17 22:24:37.993372 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:37.993333 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5bc4774985-2r78g_1cc4c446-5adf-46b0-86ce-f23ca75bb865/registry/0.log" Jul 17 22:24:38.036727 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:38.036677 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6mqj2_fe8ddec3-581d-4bd5-bfa6-0fd4d60b1694/node-ca/0.log" Jul 17 22:24:39.168561 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:39.168534 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-788f6fc9d-h598z_f8f4fcdb-b37c-471e-89f6-850451fa8af4/kube-auth-proxy/0.log" Jul 17 22:24:40.057863 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:40.057834 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kzdwp_e2d6f2a4-1393-4284-9529-3af4fdf10f71/serve-healthcheck-canary/0.log" Jul 17 22:24:40.561699 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:40.561651 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5d5nk_ad6b74e2-eda2-430c-838a-5a702016c7bd/kube-rbac-proxy/0.log" Jul 17 22:24:40.588218 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:40.588164 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5d5nk_ad6b74e2-eda2-430c-838a-5a702016c7bd/exporter/0.log" Jul 17 22:24:40.614207 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:40.614173 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5d5nk_ad6b74e2-eda2-430c-838a-5a702016c7bd/extractor/0.log" Jul 17 22:24:42.347982 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:42.347953 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-8cbj6" Jul 17 22:24:43.225221 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:43.225183 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86d8757495-sqn2m_0f885ea0-1e53-4a18-975d-be0ea8fe68ed/manager/0.log" Jul 17 22:24:43.245479 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:43.245436 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-jwq2n_3f2fc54c-9516-4d2c-b5ac-0910dca90e4b/server/0.log" Jul 17 22:24:43.340593 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:43.340560 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-54bpn_88d14f31-60b7-4d3c-bf6d-e4ff760d1dfb/manager/0.log" Jul 17 22:24:43.480637 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:43.480550 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-964b768d-r6hsf_48e2e9b0-f33c-418d-a7ce-852d2b31d50a/manager/0.log" Jul 17 22:24:43.496792 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:43.496767 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-964b768d-r6hsf_48e2e9b0-f33c-418d-a7ce-852d2b31d50a/copy-manifests/0.log" Jul 17 22:24:44.967185 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:44.967095 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7bbc9cc6cd-49q4k_2068a574-e2a0-4711-832b-7c51d9d2dabf/manager/0.log" Jul 17 22:24:45.678647 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:45.678613 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-788f6fc9d-h598z_f8f4fcdb-b37c-471e-89f6-850451fa8af4/kube-auth-proxy/0.log" Jul 17 22:24:47.131852 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:47.131813 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-788f6fc9d-h598z_f8f4fcdb-b37c-471e-89f6-850451fa8af4/kube-auth-proxy/0.log" Jul 17 22:24:51.480879 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:51.480851 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qlgx8_21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66/kube-multus-additional-cni-plugins/0.log" Jul 17 22:24:51.500008 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:51.499977 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qlgx8_21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66/egress-router-binary-copy/0.log" Jul 17 22:24:51.521180 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:51.521149 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qlgx8_21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66/cni-plugins/0.log" Jul 17 22:24:51.540220 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:51.540191 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qlgx8_21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66/bond-cni-plugin/0.log" Jul 17 22:24:51.559439 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:51.559411 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qlgx8_21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66/routeoverride-cni/0.log" Jul 17 22:24:51.579273 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:51.579249 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qlgx8_21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66/whereabouts-cni-bincopy/0.log" Jul 17 22:24:51.599194 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:51.599165 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qlgx8_21bd99ce-116a-42ab-9b1d-e8ee5d6a0a66/whereabouts-cni/0.log" Jul 17 22:24:51.634694 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:51.634660 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-b5zsn_759d3e52-70f2-4924-8b94-f06cf30df241/kube-multus/0.log" Jul 17 22:24:51.764415 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:51.764365 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d2x4m_896ed8fb-96fc-4834-83a6-807a16a340b3/network-metrics-daemon/0.log" Jul 17 22:24:51.783668 ip-10-0-130-56 kubenswrapper[2580]: I0717 22:24:51.783644 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d2x4m_896ed8fb-96fc-4834-83a6-807a16a340b3/kube-rbac-proxy/0.log"