Jul 07 14:34:26.975630 ip-10-0-142-144 systemd[1]: Starting Kubernetes Kubelet... Jul 07 14:34:27.400895 ip-10-0-142-144 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 14:34:27.400895 ip-10-0-142-144 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 14:34:27.400895 ip-10-0-142-144 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 14:34:27.400895 ip-10-0-142-144 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 14:34:27.400895 ip-10-0-142-144 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 14:34:27.400895 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.376840 2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 14:34:27.400895 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380458 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:34:27.400895 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380468 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:34:27.400895 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380472 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:34:27.400895 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380475 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:34:27.400895 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380478 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:34:27.400895 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380481 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:34:27.400895 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380484 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:34:27.400895 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380487 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:34:27.401631 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380490 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:34:27.401631 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380492 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:34:27.401631 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380495 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:34:27.401631 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380498 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:34:27.401631 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380500 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:34:27.401631 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380503 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:34:27.401631 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380505 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:34:27.401631 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380508 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:34:27.401631 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380510 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:34:27.401631 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380513 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:34:27.401631 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380515 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:34:27.401631 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380517 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:34:27.401631 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380525 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:34:27.401631 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380527 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:34:27.401631 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380530 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:34:27.401631 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380533 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:34:27.401631 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380536 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:34:27.401631 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380538 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:34:27.401631 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380542 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:34:27.402154 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380544 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:34:27.402154 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380547 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:34:27.402154 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380549 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:34:27.402154 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380552 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:34:27.402154 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380555 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:34:27.402154 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380558 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:34:27.402154 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380560 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:34:27.402154 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380563 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:34:27.402154 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380565 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:34:27.402154 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380568 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:34:27.402154 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380570 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:34:27.402154 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380573 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:34:27.402154 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380576 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:34:27.402154 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380579 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:34:27.402154 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380583 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:34:27.402154 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380587 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:34:27.402154 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380590 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:34:27.402154 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380593 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:34:27.402154 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380596 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380599 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380601 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380604 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380607 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380610 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380613 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380616 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380619 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380621 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380624 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380626 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380629 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380631 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380634 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380636 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380639 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380641 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380644 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380646 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:34:27.402712 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380649 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380651 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380654 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380656 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380659 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380661 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380664 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380667 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380669 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380672 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380675 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380678 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380680 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380682 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380685 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380687 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380691 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380694 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380696 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380699 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:34:27.403267 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380701 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380704 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380707 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380711 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380714 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380716 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380719 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380721 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380725 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380729 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380731 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380735 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380738 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.380740 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382307 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382314 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382318 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382321 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382325 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382328 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:34:27.403810 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382331 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382334 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382337 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382341 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382343 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382346 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382349 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382352 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382354 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382371 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382374 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382376 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382379 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382382 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382385 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382388 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382390 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382393 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382396 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382398 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:34:27.404511 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382401 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:34:27.405244 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382404 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:34:27.405244 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382406 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:34:27.405244 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382409 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:34:27.405244 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382412 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:34:27.405244 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382415 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:34:27.405244 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382418 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:34:27.405244 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382420 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:34:27.405244 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382423 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:34:27.405244 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382426 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:34:27.405244 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382428 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:34:27.405244 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382431 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:34:27.405244 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382434 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:34:27.405244 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382436 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:34:27.405244 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382439 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:34:27.405244 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382441 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:34:27.405244 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382445 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:34:27.405244 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382447 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:34:27.405244 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382452 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:34:27.405244 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382456 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382458 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382462 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382466 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382469 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382472 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382475 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382478 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382480 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382483 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382486 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382489 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382491 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382493 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382496 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382499 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382501 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382504 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382506 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382509 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:34:27.405909 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382512 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:34:27.406503 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382514 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:34:27.406503 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382517 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:34:27.406503 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382523 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:34:27.406503 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382526 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:34:27.406503 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382530 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:34:27.406503 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382534 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:34:27.406503 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382537 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:34:27.406503 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382540 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:34:27.406503 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382543 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:34:27.406503 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382545 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:34:27.406503 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382548 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:34:27.406503 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382550 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:34:27.406503 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382553 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:34:27.406503 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382555 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:34:27.406503 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382558 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:34:27.406503 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382560 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:34:27.406503 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382563 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:34:27.406503 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382566 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:34:27.406503 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382568 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382571 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382573 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382576 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382578 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382581 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382585 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382588 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382590 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382593 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382595 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382598 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382601 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382603 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382605 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382972 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382978 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382981 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382985 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382988 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:34:27.407034 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382991 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:34:27.407649 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382993 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:34:27.407649 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382996 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:34:27.407649 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.382998 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:34:27.407649 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383001 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:34:27.407649 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383004 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:34:27.407649 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383006 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:34:27.407649 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383009 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:34:27.407649 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383011 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:34:27.407649 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383013 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:34:27.407649 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383016 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:34:27.407649 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383018 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:34:27.407649 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383022 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:34:27.407649 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383026 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:34:27.407649 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383028 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:34:27.407649 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383031 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:34:27.407649 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383034 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:34:27.407649 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383037 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:34:27.407649 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383041 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:34:27.407649 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383044 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:34:27.408191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383048 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:34:27.408191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383052 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:34:27.408191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383055 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:34:27.408191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383058 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:34:27.408191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383061 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:34:27.408191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383063 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:34:27.408191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383066 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:34:27.408191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383069 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:34:27.408191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383073 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:34:27.408191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383075 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:34:27.408191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383078 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:34:27.408191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383080 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:34:27.408191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383083 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:34:27.408191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383085 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:34:27.408191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383088 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:34:27.408191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383091 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:34:27.408191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383093 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:34:27.408191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383096 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:34:27.408191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383098 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383101 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383104 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383106 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383109 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383111 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383114 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383116 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383119 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383121 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383124 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383126 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383130 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383133 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383135 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383138 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383141 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383144 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383147 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383150 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:34:27.408711 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383153 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383155 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383159 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383162 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383164 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383167 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383170 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383173 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383175 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383178 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383180 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383183 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383185 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383187 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383190 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383192 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383195 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383197 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383200 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383202 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:34:27.409255 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383205 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383207 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383210 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383212 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383216 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383218 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383220 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383223 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383225 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383228 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383230 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383232 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383235 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383237 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383242 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383244 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383305 2583 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383315 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383321 2583 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383325 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383332 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383337 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 14:34:27.409792 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383343 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383349 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383354 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383369 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383373 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383376 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383379 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383382 2583 flags.go:64] FLAG: --cgroup-root="" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383385 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383388 2583 flags.go:64] FLAG: --client-ca-file="" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383390 2583 flags.go:64] FLAG: --cloud-provider="external" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383393 2583 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383405 2583 flags.go:64] FLAG: --cluster-domain="" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383407 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383412 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383415 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383418 2583 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383426 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383429 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383432 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383435 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383438 2583 flags.go:64] FLAG: --contention-profiling="false" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383441 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383444 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 14:34:27.410391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383448 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383450 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383455 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383458 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383461 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383464 2583 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383467 2583 flags.go:64] FLAG: --enable-server="true" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383470 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383474 2583 flags.go:64] FLAG: --event-burst="100" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383477 2583 flags.go:64] FLAG: --event-qps="50" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383480 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383482 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383485 2583 flags.go:64] FLAG: --eviction-hard="" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383489 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383492 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383494 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383497 2583 flags.go:64] FLAG: --eviction-soft="" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383500 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383503 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383506 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383509 2583 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383512 2583 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383516 2583 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383519 2583 flags.go:64] FLAG: --feature-gates="" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383522 2583 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 14:34:27.410990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383525 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383528 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383531 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383534 2583 flags.go:64] FLAG: --healthz-port="10248" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383537 2583 flags.go:64] FLAG: --help="false" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383541 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-142-144.ec2.internal" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383543 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383548 2583 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383551 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383554 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383557 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383560 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383563 2583 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383566 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383568 2583 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383572 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383575 2583 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383578 2583 flags.go:64] FLAG: --kube-reserved="" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383580 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383583 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383586 2583 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383588 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383591 2583 flags.go:64] FLAG: --lock-file="" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383594 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 14:34:27.411970 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383596 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383599 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383605 2583 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383607 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383610 2583 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383614 2583 flags.go:64] FLAG: --logging-format="text" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383617 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383620 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383623 2583 flags.go:64] FLAG: --manifest-url="" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383625 2583 flags.go:64] FLAG: --manifest-url-header="" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383629 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383632 2583 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383636 2583 flags.go:64] FLAG: --max-pods="110" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383639 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383642 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383646 2583 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383648 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383651 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383654 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383657 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383663 2583 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383674 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383678 2583 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 14:34:27.415269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383681 2583 flags.go:64] FLAG: --pod-cidr="" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383684 2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383689 2583 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383692 2583 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383695 2583 flags.go:64] FLAG: --pods-per-core="0" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383698 2583 flags.go:64] FLAG: --port="10250" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383701 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383704 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f440cd96da59909b" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383707 2583 flags.go:64] FLAG: --qos-reserved="" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383710 2583 flags.go:64] FLAG: --read-only-port="10255" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383713 2583 flags.go:64] FLAG: --register-node="true" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383715 2583 flags.go:64] FLAG: --register-with-taints="" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383719 2583 flags.go:64] FLAG: --registry-burst="10" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383722 2583 flags.go:64] FLAG: --registry-qps="5" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383726 2583 flags.go:64] FLAG: --reserved-cpus="" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383728 2583 flags.go:64] FLAG: --reserved-memory="" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383732 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383735 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383738 2583 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383740 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383743 2583 flags.go:64] FLAG: --runonce="false" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383746 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383749 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383752 2583 flags.go:64] FLAG: --seccomp-default="false" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383755 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 14:34:27.416386 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383760 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383763 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383765 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383768 2583 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383771 2583 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383774 2583 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383776 2583 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383779 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383782 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383785 2583 flags.go:64] FLAG: --system-cgroups="" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383788 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383793 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383796 2583 flags.go:64] FLAG: --tls-cert-file="" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383798 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383802 2583 flags.go:64] FLAG: --tls-min-version="" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383805 2583 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383808 2583 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383810 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383813 2583 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383816 2583 flags.go:64] FLAG: --v="2" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383819 2583 flags.go:64] FLAG: --version="false" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383824 2583 flags.go:64] FLAG: --vmodule="" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383828 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.383831 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383951 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:34:27.418128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383956 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:34:27.419249 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383959 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:34:27.419249 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383961 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:34:27.419249 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383964 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:34:27.419249 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383966 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:34:27.419249 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383969 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:34:27.419249 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383971 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:34:27.419249 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383975 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:34:27.419249 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383978 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:34:27.419249 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383980 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:34:27.419249 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383983 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:34:27.419249 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383986 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:34:27.419249 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383988 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:34:27.419249 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383991 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:34:27.419249 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383993 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:34:27.419249 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.383996 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:34:27.419249 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384000 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:34:27.419249 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384004 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:34:27.419249 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384007 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:34:27.419249 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384010 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384012 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384015 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384017 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384020 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384022 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384025 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384027 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384030 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384033 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384036 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384038 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384041 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384045 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384048 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384051 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384054 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384057 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384060 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384063 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:34:27.420375 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384067 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384069 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384072 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384075 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384077 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384080 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384082 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384084 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384087 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384090 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384092 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384095 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384097 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384100 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384102 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384105 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384109 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384112 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384114 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384117 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:34:27.421241 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384120 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384124 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384126 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384129 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384132 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384134 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384136 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384139 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384141 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384144 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384146 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384149 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384153 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384155 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384158 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384160 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384163 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384166 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384168 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384171 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:34:27.527894 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384173 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:34:27.495089 ip-10-0-142-144 systemd[1]: Started Kubernetes Kubelet. Jul 07 14:34:27.528910 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384175 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:34:27.528910 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384178 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:34:27.528910 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384180 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:34:27.528910 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384183 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:34:27.528910 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384186 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:34:27.528910 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384188 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:34:27.528910 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384190 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:34:27.528910 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384193 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:34:27.528910 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384195 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:34:27.528910 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384198 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:34:27.528910 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384200 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:34:27.528910 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384203 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:34:27.528910 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384206 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:34:27.528910 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384208 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:34:27.528910 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384211 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:34:27.528910 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384213 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:34:27.528910 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384216 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:34:27.528910 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.384218 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:34:27.530776 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.384970 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 14:34:27.530776 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.391107 2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 14:34:27.530776 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.391122 2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 14:34:27.530776 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.391139 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 14:34:27.530776 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.391146 2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 14:34:27.530776 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391197 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:34:27.530776 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391202 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:34:27.530776 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391205 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:34:27.530776 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391209 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:34:27.530776 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391213 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:34:27.530776 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391216 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:34:27.530776 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391219 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:34:27.530776 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391222 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:34:27.530776 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391225 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391228 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391231 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391233 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391236 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391238 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391241 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391244 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391246 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391249 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391251 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391254 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391257 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391259 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391262 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391264 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391267 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391270 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391274 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391278 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:34:27.531652 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391281 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391285 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391288 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391291 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391294 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391297 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391300 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391303 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391306 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391309 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391311 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391314 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391317 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391319 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391322 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391326 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391329 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391332 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391335 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391338 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:34:27.532975 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391341 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391344 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391346 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391349 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391352 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391354 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391374 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391376 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391379 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391382 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391385 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391387 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391391 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391393 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391397 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391399 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391402 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391405 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391407 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391410 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:34:27.537689 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391413 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391415 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391417 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391420 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391423 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391426 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391428 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391431 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391434 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391436 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391439 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391441 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391444 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391446 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391449 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391451 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391454 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391456 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391459 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391461 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:34:27.538786 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391463 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:34:27.539697 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391466 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:34:27.539697 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391469 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:34:27.539697 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391471 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:34:27.539697 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391474 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:34:27.539697 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391477 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:34:27.539697 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391479 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:34:27.539697 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391482 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:34:27.539697 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391485 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:34:27.539697 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391487 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:34:27.539697 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391490 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:34:27.539697 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.391495 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 14:34:27.539697 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391600 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:34:27.539697 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391605 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:34:27.539697 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391608 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:34:27.539697 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391611 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:34:27.541231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391614 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:34:27.541231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391617 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:34:27.541231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391620 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:34:27.541231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391622 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:34:27.541231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391625 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:34:27.541231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391628 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:34:27.541231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391630 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:34:27.541231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391633 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:34:27.541231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391635 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:34:27.541231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391639 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:34:27.541231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391643 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:34:27.541231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391646 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:34:27.541231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391650 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:34:27.541231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391652 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:34:27.541231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391655 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:34:27.541231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391658 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:34:27.541231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391660 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:34:27.541231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391663 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:34:27.541231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391676 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391679 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391682 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391684 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391687 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391690 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391693 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391696 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391700 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391702 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391705 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391709 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391711 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391714 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391716 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391719 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391721 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391724 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391726 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391728 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:34:27.542128 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391731 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391733 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391736 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391738 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391741 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391743 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391745 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391748 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391750 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391753 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391756 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391758 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391763 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391766 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391768 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391770 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391773 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391776 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391778 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391781 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:34:27.542974 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391784 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:34:27.545178 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391786 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:34:27.545178 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391789 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:34:27.545178 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391793 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:34:27.545178 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391795 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:34:27.545178 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391798 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:34:27.545178 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391800 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:34:27.545178 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391803 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:34:27.545178 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391805 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:34:27.545178 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391807 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:34:27.545178 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391810 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:34:27.545178 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391812 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:34:27.545178 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391815 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:34:27.545178 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391817 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:34:27.545178 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391820 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:34:27.545178 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391823 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:34:27.545178 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391825 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:34:27.545178 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391827 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:34:27.545178 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391831 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:34:27.545178 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391835 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:34:27.546049 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391837 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:34:27.546049 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391840 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:34:27.546049 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391843 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:34:27.546049 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391845 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:34:27.546049 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391849 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:34:27.546049 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391851 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:34:27.546049 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391854 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:34:27.546049 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391856 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:34:27.546049 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391859 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:34:27.546049 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391861 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:34:27.546049 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391864 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:34:27.546049 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391866 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:34:27.546049 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391869 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:34:27.546049 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391871 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:34:27.546049 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391874 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:34:27.546049 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391878 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:34:27.546049 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:27.391881 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:34:27.547587 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.391886 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 14:34:27.547587 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.391973 2583 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 14:34:27.547587 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.394768 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 14:34:27.547587 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.396340 2583 server.go:1019] "Starting client certificate rotation" Jul 07 14:34:27.547587 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.396448 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 14:34:27.547587 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.400070 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 14:34:27.547587 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.400185 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 14:34:27.547587 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.407217 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 14:34:27.547587 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.432420 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 14:34:27.547587 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.442596 2583 log.go:25] "Validated CRI v1 runtime API" Jul 07 14:34:27.547587 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.445513 2583 log.go:25] "Validated CRI v1 image API" Jul 07 14:34:27.547587 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.446228 2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 14:34:27.548245 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.449666 2583 fs.go:135] Filesystem UUIDs: map[352a06d8-79ec-43f0-b8ae-c520719c30d5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 be453716-4ca2-4794-8681-1507f7c5005c:/dev/nvme0n1p3] Jul 07 14:34:27.548245 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.449679 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 14:34:27.638026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.454183 2583 manager.go:216] Machine: {Timestamp:2026-07-07 14:34:27.453736859 +0000 UTC m=+0.364930530 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2499996 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec221f5c21e0c9c66a5b0a76d70c04d9 SystemUUID:ec221f5c-21e0-c9c6-6a5b-0a76d70c04d9 BootID:320793a6-40e1-4205-bc90-694bd32315e5 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e2:f4:2f:4d:8f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e2:f4:2f:4d:8f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:07:f3:66:42:2e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 14:34:27.638026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.454271 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 14:34:27.638026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.454332 2583 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 14:34:27.638026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.459393 2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 14:34:27.638026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.459416 2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-144.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 14:34:27.638026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.460197 2583 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 14:34:27.638026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.460205 2583 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 14:34:27.638026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.460216 2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 14:34:27.638026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.461050 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 14:34:27.638026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.462321 2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 14:34:27.638026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.465678 2583 state_mem.go:36] "Initialized new in-memory state store" Jul 07 14:34:27.638026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.465787 2583 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 14:34:27.638026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.465812 2583 kubelet.go:486] "Attempting to sync node with API server" Jul 07 14:34:27.638026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.465821 2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 14:34:27.638026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.465835 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 14:34:27.638026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.465843 2583 kubelet.go:398] "Adding apiserver pod source" Jul 07 14:34:27.638026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.465851 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 14:34:27.638026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.466165 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.466180 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.473957 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z5scs" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:27.475818 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-144.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:27.475843 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.480673 2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.480948 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z5scs" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.483089 2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.483117 2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.489670 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.489686 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.489693 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.489708 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.489717 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.489722 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.489730 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.489735 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.489740 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.489745 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.489759 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.489764 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.491381 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.491389 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.493988 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-144.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.494151 2583 server.go:1268] "Started kubelet" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.494542 2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.494960 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.495034 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.496453 2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.497706 2583 server.go:310] "Adding debug handlers to kubelet server" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.505418 2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.505471 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:27.505660 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-144.ec2.internal\" not found" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:27.506026 2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.506105 2583 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.506116 2583 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.506104 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.506231 2583 reconstruct.go:97] "Volume reconstruction finished" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.506240 2583 reconciler.go:29] "Reconciler: start to sync state" Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.506265 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.506277 2583 factory.go:55] Registering systemd factory Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.506285 2583 factory.go:223] Registration of the systemd container factory successfully Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.506480 2583 factory.go:153] Registering CRI-O factory Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.506491 2583 factory.go:223] Registration of the crio container factory successfully Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.506509 2583 factory.go:103] Registering Raw factory Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.506522 2583 manager.go:1196] Started watching for new ooms in manager Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.506898 2583 manager.go:349] Starting recovery of all containers Jul 07 14:34:27.639009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.512571 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 14:34:27.641003 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.515301 2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:34:27.641003 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.515887 2583 manager.go:354] Recovery completed Jul 07 14:34:27.641003 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:27.519112 2583 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-144.ec2.internal\" not found" node="ip-10-0-142-144.ec2.internal" Jul 07 14:34:27.641003 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.520495 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:34:27.641003 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.522756 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:34:27.641003 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.522782 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:34:27.641003 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.522795 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:34:27.641003 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.523233 2583 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 14:34:27.641003 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.523242 2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 14:34:27.641003 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.523257 2583 state_mem.go:36] "Initialized new in-memory state store" Jul 07 14:34:27.641003 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.523235 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:34:27.641003 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.523895 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:34:27.641003 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.523916 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:34:27.641003 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.523927 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:34:27.641003 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.554463 2583 policy_none.go:49] "None policy: Start" Jul 07 14:34:27.641003 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.554479 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 14:34:27.641003 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.554495 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 14:34:27.641003 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.555991 2583 policy_none.go:47] "Start" Jul 07 14:34:27.641003 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:27.605904 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-144.ec2.internal\" not found" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.658984 2583 manager.go:337] "Starting Device Plugin manager" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:27.659155 2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.659168 2583 server.go:87] "Starting device plugin registration server" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.659400 2583 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.659413 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.659497 2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.659586 2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.659596 2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:27.660530 2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:27.660567 2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-144.ec2.internal\" not found" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.760196 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.761097 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.761127 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.761137 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.761166 2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-144.ec2.internal" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.768439 2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-144.ec2.internal" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:27.768456 2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-144.ec2.internal\": node \"ip-10-0-142-144.ec2.internal\" not found" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:27.780483 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-144.ec2.internal\" not found" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.785792 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.785829 2583 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.785850 2583 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:27.785880 2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 14:34:27.852198 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.788036 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:27.881599 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-144.ec2.internal\" not found" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.886020 2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal"] Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.886093 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.888802 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.888828 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.888841 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.890437 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.890572 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.890610 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.891159 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.891180 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.891190 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.891221 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.891240 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.891251 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.892485 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.892515 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.893282 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.893309 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.893325 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.907750 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f554e2b8de4942207b69bb5df078360-config\") pod \"kube-apiserver-proxy-ip-10-0-142-144.ec2.internal\" (UID: \"2f554e2b8de4942207b69bb5df078360\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.907773 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b14e6cdbc81539fdf89e4bd7f5a99a19-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal\" (UID: \"b14e6cdbc81539fdf89e4bd7f5a99a19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:27.907792 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b14e6cdbc81539fdf89e4bd7f5a99a19-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal\" (UID: \"b14e6cdbc81539fdf89e4bd7f5a99a19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:27.910976 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-144.ec2.internal\" not found" node="ip-10-0-142-144.ec2.internal" Jul 07 14:34:27.991350 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:27.913170 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-144.ec2.internal\" not found" node="ip-10-0-142-144.ec2.internal" Jul 07 14:34:28.209200 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:28.011097 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-144.ec2.internal\" not found" Jul 07 14:34:28.209200 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.011127 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b14e6cdbc81539fdf89e4bd7f5a99a19-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal\" (UID: \"b14e6cdbc81539fdf89e4bd7f5a99a19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" Jul 07 14:34:28.209200 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.011157 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f554e2b8de4942207b69bb5df078360-config\") pod \"kube-apiserver-proxy-ip-10-0-142-144.ec2.internal\" (UID: \"2f554e2b8de4942207b69bb5df078360\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal" Jul 07 14:34:28.209200 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.011175 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b14e6cdbc81539fdf89e4bd7f5a99a19-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal\" (UID: \"b14e6cdbc81539fdf89e4bd7f5a99a19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" Jul 07 14:34:28.209200 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.011221 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b14e6cdbc81539fdf89e4bd7f5a99a19-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal\" (UID: \"b14e6cdbc81539fdf89e4bd7f5a99a19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" Jul 07 14:34:28.209200 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.011236 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f554e2b8de4942207b69bb5df078360-config\") pod \"kube-apiserver-proxy-ip-10-0-142-144.ec2.internal\" (UID: \"2f554e2b8de4942207b69bb5df078360\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal" Jul 07 14:34:28.209200 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.011236 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b14e6cdbc81539fdf89e4bd7f5a99a19-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal\" (UID: \"b14e6cdbc81539fdf89e4bd7f5a99a19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" Jul 07 14:34:28.209200 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:28.111464 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-144.ec2.internal\" not found" Jul 07 14:34:28.214013 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:28.212307 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-144.ec2.internal\" not found" Jul 07 14:34:28.214784 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.214640 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" Jul 07 14:34:28.362827 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.216975 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal" Jul 07 14:34:28.362827 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:28.312387 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-144.ec2.internal\" not found" Jul 07 14:34:28.362827 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.361580 2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:34:28.569360 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.397631 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 14:34:28.569360 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.397731 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 14:34:28.569360 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.397756 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 14:34:28.569360 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.397758 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 14:34:28.569360 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:28.413091 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-144.ec2.internal\" not found" Jul 07 14:34:28.569360 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.474712 2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:34:28.569360 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.486142 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 14:29:28 +0000 UTC" deadline="2027-12-31 17:17:25.241835375 +0000 UTC" Jul 07 14:34:28.569360 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.486183 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13010h42m56.75565674s" Jul 07 14:34:28.569360 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.506260 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 14:34:28.569360 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.506286 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" Jul 07 14:34:28.569360 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.517624 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 14:34:28.569360 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.518768 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal" Jul 07 14:34:28.569360 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.519992 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 14:34:28.569360 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.527826 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 14:34:28.569360 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.544660 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fdp8c" Jul 07 14:34:28.569360 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.554307 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fdp8c" Jul 07 14:34:28.990828 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:28.990799 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f554e2b8de4942207b69bb5df078360.slice/crio-54f1e9f8e823e403587913249acb7785a4686a34dc8a2dee8d1846c49b360ebe WatchSource:0}: Error finding container 54f1e9f8e823e403587913249acb7785a4686a34dc8a2dee8d1846c49b360ebe: Status 404 returned error can't find the container with id 54f1e9f8e823e403587913249acb7785a4686a34dc8a2dee8d1846c49b360ebe Jul 07 14:34:28.991324 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:28.991301 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14e6cdbc81539fdf89e4bd7f5a99a19.slice/crio-86e9e103f53244cce26e7c3755273d1b87952ab124f3a85767dfca9523f3115d WatchSource:0}: Error finding container 86e9e103f53244cce26e7c3755273d1b87952ab124f3a85767dfca9523f3115d: Status 404 returned error can't find the container with id 86e9e103f53244cce26e7c3755273d1b87952ab124f3a85767dfca9523f3115d Jul 07 14:34:28.996513 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:28.996498 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:34:29.402568 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.402545 2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:34:29.467947 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.467924 2583 apiserver.go:52] "Watching apiserver" Jul 07 14:34:29.475644 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.475613 2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 14:34:29.475992 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.475964 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-sjd5c","kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal","openshift-cluster-node-tuning-operator/tuned-b8dkc","openshift-multus/network-metrics-daemon-xsfqs","openshift-network-diagnostics/network-check-target-8d4jp","openshift-network-operator/iptables-alerter-6hkgq","openshift-ovn-kubernetes/ovnkube-node-5r6r4","kube-system/konnectivity-agent-2rp28","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n","openshift-dns/node-resolver-rzkvv","openshift-image-registry/node-ca-h78mx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal","openshift-multus/multus-additional-cni-plugins-pn6lw","openshift-multus/multus-jd2vc"] Jul 07 14:34:29.478640 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.478617 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.479978 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.479957 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.481020 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.480998 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 14:34:29.481353 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.481333 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 14:34:29.481476 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.481387 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 14:34:29.481476 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.481391 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 14:34:29.481598 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.481476 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.481710 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.481683 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 14:34:29.481832 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.481799 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9kqrk\"" Jul 07 14:34:29.482487 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.482336 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 14:34:29.482487 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.482377 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 14:34:29.482487 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.482456 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f62zn\"" Jul 07 14:34:29.483155 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.482733 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 14:34:29.483155 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.482806 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 14:34:29.483155 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.482869 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:29.483155 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.482892 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 14:34:29.483155 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:29.482936 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:34:29.483155 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.483030 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 14:34:29.483626 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.483605 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2sk6v\"" Jul 07 14:34:29.484002 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.483978 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:34:29.484117 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.483988 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 14:34:29.484471 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.484448 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:29.484578 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:29.484516 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d4jp" podUID="b7a37935-544f-4f5d-8392-2ef44c031582" Jul 07 14:34:29.485950 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.485928 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6hkgq" Jul 07 14:34:29.487300 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.487262 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.487880 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.487855 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:34:29.488568 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.488108 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 14:34:29.488568 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.488136 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 14:34:29.488568 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.488166 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rxtbv\"" Jul 07 14:34:29.488775 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.488717 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:29.488851 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:29.488775 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjd5c" podUID="3a0f179d-66e3-41bd-aa10-71efe8001185" Jul 07 14:34:29.489349 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.489323 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 14:34:29.489475 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.489369 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 14:34:29.489475 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.489327 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sltn4\"" Jul 07 14:34:29.489597 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.489521 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 14:34:29.490213 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.490196 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.491617 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.491598 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rzkvv" Jul 07 14:34:29.492184 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.492165 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-clmnv\"" Jul 07 14:34:29.492256 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.492173 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 14:34:29.493549 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.493528 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q68z7\"" Jul 07 14:34:29.493803 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.493784 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h78mx" Jul 07 14:34:29.493896 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.493877 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 14:34:29.493946 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.493914 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 14:34:29.495010 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.494982 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2rp28" Jul 07 14:34:29.496014 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.495993 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 14:34:29.496121 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.496036 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 14:34:29.496121 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.496064 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zrghk\"" Jul 07 14:34:29.496121 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.496091 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 14:34:29.497484 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.497461 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m5k5g\"" Jul 07 14:34:29.497832 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.497814 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 14:34:29.498735 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.498701 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 14:34:29.507328 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.507308 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 14:34:29.519785 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.519764 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95lfm\" (UniqueName: \"kubernetes.io/projected/7fcf42de-dcd6-4569-9eb9-3e3a53c101f0-kube-api-access-95lfm\") pod \"iptables-alerter-6hkgq\" (UID: \"7fcf42de-dcd6-4569-9eb9-3e3a53c101f0\") " pod="openshift-network-operator/iptables-alerter-6hkgq" Jul 07 14:34:29.519897 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.519790 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-cnibin\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.519897 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.519811 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a0f179d-66e3-41bd-aa10-71efe8001185-dbus\") pod \"global-pull-secret-syncer-sjd5c\" (UID: \"3a0f179d-66e3-41bd-aa10-71efe8001185\") " pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:29.519897 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.519835 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b602982-27d1-4239-b9c7-7e09013cf3d5-cni-binary-copy\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.519897 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.519858 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-host-var-lib-cni-bin\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.520052 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.519916 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-cni-netd\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.520052 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.519955 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4jm8\" (UniqueName: \"kubernetes.io/projected/e8c3c836-a810-43f4-bed4-c66cc24e365f-kube-api-access-t4jm8\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.520052 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.519987 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80308b12-3ccc-477c-9578-1ac78d11db32-serviceca\") pod \"node-ca-h78mx\" (UID: \"80308b12-3ccc-477c-9578-1ac78d11db32\") " pod="openshift-image-registry/node-ca-h78mx" Jul 07 14:34:29.520052 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520014 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ded5607-54b2-4314-bb3f-443bed8150e6-agent-certs\") pod \"konnectivity-agent-2rp28\" (UID: \"2ded5607-54b2-4314-bb3f-443bed8150e6\") " pod="kube-system/konnectivity-agent-2rp28" Jul 07 14:34:29.520052 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520038 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-etc-openvswitch\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.520315 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520081 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-log-socket\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.520315 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520115 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-kubernetes\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.520315 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520141 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-socket-dir\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.520315 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520166 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-host-var-lib-kubelet\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.520315 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520203 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-host-run-multus-certs\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.520315 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520227 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-node-log\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.520315 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520248 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e8c3c836-a810-43f4-bed4-c66cc24e365f-ovnkube-config\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.520315 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520265 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8qjv\" (UniqueName: \"kubernetes.io/projected/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-kube-api-access-q8qjv\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.520315 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520281 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kprh\" (UniqueName: \"kubernetes.io/projected/80308b12-3ccc-477c-9578-1ac78d11db32-kube-api-access-2kprh\") pod \"node-ca-h78mx\" (UID: \"80308b12-3ccc-477c-9578-1ac78d11db32\") " pod="openshift-image-registry/node-ca-h78mx" Jul 07 14:34:29.520315 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520299 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-sysctl-d\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.520798 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520323 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-sysctl-conf\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.520798 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520343 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-systemd\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.520798 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520357 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-hostroot\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.520798 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520379 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f877ac1-ec8e-437e-a186-28b499cf63e1-tmp-dir\") pod \"node-resolver-rzkvv\" (UID: \"1f877ac1-ec8e-437e-a186-28b499cf63e1\") " pod="openshift-dns/node-resolver-rzkvv" Jul 07 14:34:29.520798 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520416 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80308b12-3ccc-477c-9578-1ac78d11db32-host\") pod \"node-ca-h78mx\" (UID: \"80308b12-3ccc-477c-9578-1ac78d11db32\") " pod="openshift-image-registry/node-ca-h78mx" Jul 07 14:34:29.520798 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520442 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8945047-a8e7-4958-9133-a5af6e9ac2b1-cnibin\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.520798 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520458 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwlvv\" (UniqueName: \"kubernetes.io/projected/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-kube-api-access-jwlvv\") pod \"network-metrics-daemon-xsfqs\" (UID: \"d60cabb9-29c9-4f9d-b147-2ca1844f5d96\") " pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:29.520798 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520530 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5hbv\" (UniqueName: \"kubernetes.io/projected/1f877ac1-ec8e-437e-a186-28b499cf63e1-kube-api-access-k5hbv\") pod \"node-resolver-rzkvv\" (UID: \"1f877ac1-ec8e-437e-a186-28b499cf63e1\") " pod="openshift-dns/node-resolver-rzkvv" Jul 07 14:34:29.520798 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520579 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-sys\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.520798 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520605 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-var-lib-kubelet\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.520798 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520636 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.520798 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520683 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-registration-dir\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.520798 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520739 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cltf\" (UniqueName: \"kubernetes.io/projected/4b602982-27d1-4239-b9c7-7e09013cf3d5-kube-api-access-9cltf\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.520798 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520765 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-run-netns\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.520798 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520789 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-host\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.521467 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520820 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-tuned\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.521467 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520844 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-multus-conf-dir\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.521467 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520868 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-cni-bin\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.521467 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520901 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjwxh\" (UniqueName: \"kubernetes.io/projected/a8945047-a8e7-4958-9133-a5af6e9ac2b1-kube-api-access-zjwxh\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.521467 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520932 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f877ac1-ec8e-437e-a186-28b499cf63e1-hosts-file\") pod \"node-resolver-rzkvv\" (UID: \"1f877ac1-ec8e-437e-a186-28b499cf63e1\") " pod="openshift-dns/node-resolver-rzkvv" Jul 07 14:34:29.521467 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.520983 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-run-systemd\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.521467 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521004 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-var-lib-openvswitch\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.521467 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521052 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8945047-a8e7-4958-9133-a5af6e9ac2b1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.521467 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521091 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8945047-a8e7-4958-9133-a5af6e9ac2b1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.521467 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521117 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-run\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.521467 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521142 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-device-dir\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.521467 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521169 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret\") pod \"global-pull-secret-syncer-sjd5c\" (UID: \"3a0f179d-66e3-41bd-aa10-71efe8001185\") " pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:29.521467 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521204 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-system-cni-dir\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.521467 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521237 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-host-run-k8s-cni-cncf-io\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.521467 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521262 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-host-run-netns\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.521467 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521289 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-systemd-units\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.522165 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521314 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-slash\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.522165 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521342 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-sysconfig\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.522165 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521367 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-run-openvswitch\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.522165 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521414 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8945047-a8e7-4958-9133-a5af6e9ac2b1-system-cni-dir\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.522165 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521447 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8945047-a8e7-4958-9133-a5af6e9ac2b1-cni-binary-copy\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.522165 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521477 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs\") pod \"network-metrics-daemon-xsfqs\" (UID: \"d60cabb9-29c9-4f9d-b147-2ca1844f5d96\") " pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:29.522165 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521504 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-sys-fs\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.522165 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521562 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqdjc\" (UniqueName: \"kubernetes.io/projected/c8883f2e-855d-4c69-be5e-2a09c3e9686d-kube-api-access-dqdjc\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.522165 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521584 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a0f179d-66e3-41bd-aa10-71efe8001185-kubelet-config\") pod \"global-pull-secret-syncer-sjd5c\" (UID: \"3a0f179d-66e3-41bd-aa10-71efe8001185\") " pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:29.522165 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521610 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-os-release\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.522165 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521633 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-kubelet\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.522165 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521648 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-run-ovn\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.522165 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521674 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-multus-cni-dir\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.522165 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521690 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-multus-socket-dir-parent\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.522165 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521728 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.522165 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521759 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8c3c836-a810-43f4-bed4-c66cc24e365f-ovn-node-metrics-cert\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.522852 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521786 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pptp\" (UniqueName: \"kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp\") pod \"network-check-target-8d4jp\" (UID: \"b7a37935-544f-4f5d-8392-2ef44c031582\") " pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:29.522852 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521813 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-etc-kubernetes\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.522852 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521837 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ded5607-54b2-4314-bb3f-443bed8150e6-konnectivity-ca\") pod \"konnectivity-agent-2rp28\" (UID: \"2ded5607-54b2-4314-bb3f-443bed8150e6\") " pod="kube-system/konnectivity-agent-2rp28" Jul 07 14:34:29.522852 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521888 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-modprobe-d\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.522852 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521916 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8883f2e-855d-4c69-be5e-2a09c3e9686d-tmp\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.522852 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521941 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7fcf42de-dcd6-4569-9eb9-3e3a53c101f0-iptables-alerter-script\") pod \"iptables-alerter-6hkgq\" (UID: \"7fcf42de-dcd6-4569-9eb9-3e3a53c101f0\") " pod="openshift-network-operator/iptables-alerter-6hkgq" Jul 07 14:34:29.522852 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.521977 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-etc-selinux\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.522852 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.522013 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-host-var-lib-cni-multus\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.522852 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.522031 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-run-ovn-kubernetes\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.522852 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.522070 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e8c3c836-a810-43f4-bed4-c66cc24e365f-ovnkube-script-lib\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.522852 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.522088 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fcf42de-dcd6-4569-9eb9-3e3a53c101f0-host-slash\") pod \"iptables-alerter-6hkgq\" (UID: \"7fcf42de-dcd6-4569-9eb9-3e3a53c101f0\") " pod="openshift-network-operator/iptables-alerter-6hkgq" Jul 07 14:34:29.522852 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.522102 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b602982-27d1-4239-b9c7-7e09013cf3d5-multus-daemon-config\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.522852 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.522123 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e8c3c836-a810-43f4-bed4-c66cc24e365f-env-overrides\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.522852 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.522162 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a8945047-a8e7-4958-9133-a5af6e9ac2b1-os-release\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.522852 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.522191 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8945047-a8e7-4958-9133-a5af6e9ac2b1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.522852 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.522218 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-lib-modules\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.556185 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.556135 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:29:29 +0000 UTC" deadline="2027-12-04 21:15:06.923201932 +0000 UTC" Jul 07 14:34:29.556303 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.556185 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12366h40m37.367021446s" Jul 07 14:34:29.622781 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.622748 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-socket-dir\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.622781 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.622779 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-host-var-lib-kubelet\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.623333 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.622835 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-host-var-lib-kubelet\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.623333 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.622871 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-host-run-multus-certs\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.623333 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.622897 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-node-log\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.623333 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.622920 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-host-run-multus-certs\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.623333 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.622919 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-socket-dir\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.623333 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.622924 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e8c3c836-a810-43f4-bed4-c66cc24e365f-ovnkube-config\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.623333 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.622953 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-node-log\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.623333 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.622984 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8qjv\" (UniqueName: \"kubernetes.io/projected/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-kube-api-access-q8qjv\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.623333 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623022 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kprh\" (UniqueName: \"kubernetes.io/projected/80308b12-3ccc-477c-9578-1ac78d11db32-kube-api-access-2kprh\") pod \"node-ca-h78mx\" (UID: \"80308b12-3ccc-477c-9578-1ac78d11db32\") " pod="openshift-image-registry/node-ca-h78mx" Jul 07 14:34:29.623333 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623217 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-sysctl-d\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.623333 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623268 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-sysctl-conf\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.623333 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623292 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-systemd\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.623333 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623314 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-hostroot\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.623333 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623338 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f877ac1-ec8e-437e-a186-28b499cf63e1-tmp-dir\") pod \"node-resolver-rzkvv\" (UID: \"1f877ac1-ec8e-437e-a186-28b499cf63e1\") " pod="openshift-dns/node-resolver-rzkvv" Jul 07 14:34:29.624114 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623362 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80308b12-3ccc-477c-9578-1ac78d11db32-host\") pod \"node-ca-h78mx\" (UID: \"80308b12-3ccc-477c-9578-1ac78d11db32\") " pod="openshift-image-registry/node-ca-h78mx" Jul 07 14:34:29.624114 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623385 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8945047-a8e7-4958-9133-a5af6e9ac2b1-cnibin\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.624114 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623429 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwlvv\" (UniqueName: \"kubernetes.io/projected/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-kube-api-access-jwlvv\") pod \"network-metrics-daemon-xsfqs\" (UID: \"d60cabb9-29c9-4f9d-b147-2ca1844f5d96\") " pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:29.624114 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623452 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-sysctl-d\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.624114 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623455 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5hbv\" (UniqueName: \"kubernetes.io/projected/1f877ac1-ec8e-437e-a186-28b499cf63e1-kube-api-access-k5hbv\") pod \"node-resolver-rzkvv\" (UID: \"1f877ac1-ec8e-437e-a186-28b499cf63e1\") " pod="openshift-dns/node-resolver-rzkvv" Jul 07 14:34:29.624114 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623479 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e8c3c836-a810-43f4-bed4-c66cc24e365f-ovnkube-config\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.624114 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623504 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-sys\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.624114 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623534 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-var-lib-kubelet\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.624114 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623562 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.624114 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623587 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-registration-dir\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.624114 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623636 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cltf\" (UniqueName: \"kubernetes.io/projected/4b602982-27d1-4239-b9c7-7e09013cf3d5-kube-api-access-9cltf\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.624114 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623653 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-sysctl-conf\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.624114 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623661 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-run-netns\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.624114 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623687 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-host\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.624114 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623712 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-tuned\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.624114 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623715 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-hostroot\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.624114 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623739 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-multus-conf-dir\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.624114 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623765 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8945047-a8e7-4958-9133-a5af6e9ac2b1-cnibin\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.625030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623767 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-cni-bin\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.625030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623797 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjwxh\" (UniqueName: \"kubernetes.io/projected/a8945047-a8e7-4958-9133-a5af6e9ac2b1-kube-api-access-zjwxh\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.625030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623807 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-cni-bin\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.625030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623817 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f877ac1-ec8e-437e-a186-28b499cf63e1-hosts-file\") pod \"node-resolver-rzkvv\" (UID: \"1f877ac1-ec8e-437e-a186-28b499cf63e1\") " pod="openshift-dns/node-resolver-rzkvv" Jul 07 14:34:29.625030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623843 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-run-systemd\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.625030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623858 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-sys\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.625030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623888 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80308b12-3ccc-477c-9578-1ac78d11db32-host\") pod \"node-ca-h78mx\" (UID: \"80308b12-3ccc-477c-9578-1ac78d11db32\") " pod="openshift-image-registry/node-ca-h78mx" Jul 07 14:34:29.625030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623899 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-var-lib-openvswitch\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.625030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623966 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-var-lib-kubelet\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.625030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624011 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.625030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624026 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f877ac1-ec8e-437e-a186-28b499cf63e1-hosts-file\") pod \"node-resolver-rzkvv\" (UID: \"1f877ac1-ec8e-437e-a186-28b499cf63e1\") " pod="openshift-dns/node-resolver-rzkvv" Jul 07 14:34:29.625030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624061 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-registration-dir\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.625030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624070 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-run-systemd\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.625030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624177 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f877ac1-ec8e-437e-a186-28b499cf63e1-tmp-dir\") pod \"node-resolver-rzkvv\" (UID: \"1f877ac1-ec8e-437e-a186-28b499cf63e1\") " pod="openshift-dns/node-resolver-rzkvv" Jul 07 14:34:29.625030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624256 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-systemd\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.625030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624298 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-run-netns\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.625030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.623862 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-var-lib-openvswitch\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.625030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624300 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-multus-conf-dir\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.625941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624342 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8945047-a8e7-4958-9133-a5af6e9ac2b1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.625941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624342 2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 14:34:29.625941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624368 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-host\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.625941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624542 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8945047-a8e7-4958-9133-a5af6e9ac2b1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.625941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624370 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8945047-a8e7-4958-9133-a5af6e9ac2b1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.625941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624586 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-run\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.625941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624632 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-device-dir\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.625941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624656 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret\") pod \"global-pull-secret-syncer-sjd5c\" (UID: \"3a0f179d-66e3-41bd-aa10-71efe8001185\") " pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:29.625941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624699 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-system-cni-dir\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.625941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624721 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-host-run-k8s-cni-cncf-io\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.625941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624751 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-host-run-netns\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.625941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624770 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-systemd-units\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.625941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624797 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-slash\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.625941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624823 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-sysconfig\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.625941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624843 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-run-openvswitch\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.625941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624866 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8945047-a8e7-4958-9133-a5af6e9ac2b1-system-cni-dir\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.625941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624885 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8945047-a8e7-4958-9133-a5af6e9ac2b1-cni-binary-copy\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.626749 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624905 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs\") pod \"network-metrics-daemon-xsfqs\" (UID: \"d60cabb9-29c9-4f9d-b147-2ca1844f5d96\") " pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:29.626749 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624932 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-sys-fs\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.626749 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624965 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqdjc\" (UniqueName: \"kubernetes.io/projected/c8883f2e-855d-4c69-be5e-2a09c3e9686d-kube-api-access-dqdjc\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.626749 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624986 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a0f179d-66e3-41bd-aa10-71efe8001185-kubelet-config\") pod \"global-pull-secret-syncer-sjd5c\" (UID: \"3a0f179d-66e3-41bd-aa10-71efe8001185\") " pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:29.626749 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625009 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-os-release\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.626749 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625029 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-kubelet\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.626749 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625052 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-run-ovn\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.626749 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625090 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-multus-cni-dir\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.626749 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625114 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-multus-socket-dir-parent\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.626749 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625140 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.626749 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625171 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8c3c836-a810-43f4-bed4-c66cc24e365f-ovn-node-metrics-cert\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.626749 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625196 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pptp\" (UniqueName: \"kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp\") pod \"network-check-target-8d4jp\" (UID: \"b7a37935-544f-4f5d-8392-2ef44c031582\") " pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:29.626749 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625226 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-etc-kubernetes\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.626749 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:29.625238 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:29.626749 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625269 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ded5607-54b2-4314-bb3f-443bed8150e6-konnectivity-ca\") pod \"konnectivity-agent-2rp28\" (UID: \"2ded5607-54b2-4314-bb3f-443bed8150e6\") " pod="kube-system/konnectivity-agent-2rp28" Jul 07 14:34:29.626749 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:29.625327 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs podName:d60cabb9-29c9-4f9d-b147-2ca1844f5d96 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:30.125291264 +0000 UTC m=+3.036484938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs") pod "network-metrics-daemon-xsfqs" (UID: "d60cabb9-29c9-4f9d-b147-2ca1844f5d96") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:29.626749 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625348 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-modprobe-d\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.627559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625354 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-system-cni-dir\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.627559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625377 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8883f2e-855d-4c69-be5e-2a09c3e9686d-tmp\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.627559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625407 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7fcf42de-dcd6-4569-9eb9-3e3a53c101f0-iptables-alerter-script\") pod \"iptables-alerter-6hkgq\" (UID: \"7fcf42de-dcd6-4569-9eb9-3e3a53c101f0\") " pod="openshift-network-operator/iptables-alerter-6hkgq" Jul 07 14:34:29.627559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625455 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-etc-selinux\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.627559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625480 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-host-var-lib-cni-multus\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.627559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625499 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-run\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.627559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625506 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-run-ovn-kubernetes\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.627559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625527 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-device-dir\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.627559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625549 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-run-ovn-kubernetes\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.627559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.624929 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8945047-a8e7-4958-9133-a5af6e9ac2b1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.627559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625554 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e8c3c836-a810-43f4-bed4-c66cc24e365f-ovnkube-script-lib\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.627559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625592 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-sysconfig\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.627559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625604 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fcf42de-dcd6-4569-9eb9-3e3a53c101f0-host-slash\") pod \"iptables-alerter-6hkgq\" (UID: \"7fcf42de-dcd6-4569-9eb9-3e3a53c101f0\") " pod="openshift-network-operator/iptables-alerter-6hkgq" Jul 07 14:34:29.627559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625607 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-sys-fs\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.627559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625638 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8945047-a8e7-4958-9133-a5af6e9ac2b1-system-cni-dir\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.627559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.625644 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-host-run-netns\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.627559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.626455 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-os-release\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.627559 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:29.626577 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:29.628407 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:29.626645 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret podName:3a0f179d-66e3-41bd-aa10-71efe8001185 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:30.126618815 +0000 UTC m=+3.037812482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret") pod "global-pull-secret-syncer-sjd5c" (UID: "3a0f179d-66e3-41bd-aa10-71efe8001185") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:29.628407 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.626698 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-kubelet\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.628407 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.626701 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-slash\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.628407 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.626760 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b602982-27d1-4239-b9c7-7e09013cf3d5-multus-daemon-config\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.628407 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.626805 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e8c3c836-a810-43f4-bed4-c66cc24e365f-env-overrides\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.628407 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.626842 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-run-ovn\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.628407 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.626843 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a8945047-a8e7-4958-9133-a5af6e9ac2b1-os-release\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.628407 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.626910 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a8945047-a8e7-4958-9133-a5af6e9ac2b1-os-release\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.628407 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.626804 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a0f179d-66e3-41bd-aa10-71efe8001185-kubelet-config\") pod \"global-pull-secret-syncer-sjd5c\" (UID: \"3a0f179d-66e3-41bd-aa10-71efe8001185\") " pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:29.628407 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.626997 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-multus-socket-dir-parent\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.628407 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.627060 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.628407 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.627113 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e8c3c836-a810-43f4-bed4-c66cc24e365f-ovnkube-script-lib\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.628407 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.627196 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-host-run-k8s-cni-cncf-io\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.628407 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.627689 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-multus-cni-dir\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.628407 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.627765 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-systemd-units\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.628407 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.627764 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b602982-27d1-4239-b9c7-7e09013cf3d5-multus-daemon-config\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.628407 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.627908 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-etc-selinux\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.629221 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.627948 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8945047-a8e7-4958-9133-a5af6e9ac2b1-cni-binary-copy\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.629221 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.628174 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-etc-kubernetes\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.629221 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.628510 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7fcf42de-dcd6-4569-9eb9-3e3a53c101f0-iptables-alerter-script\") pod \"iptables-alerter-6hkgq\" (UID: \"7fcf42de-dcd6-4569-9eb9-3e3a53c101f0\") " pod="openshift-network-operator/iptables-alerter-6hkgq" Jul 07 14:34:29.629221 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.628577 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-host-var-lib-cni-multus\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.629221 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.628514 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-modprobe-d\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.629221 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.628594 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fcf42de-dcd6-4569-9eb9-3e3a53c101f0-host-slash\") pod \"iptables-alerter-6hkgq\" (UID: \"7fcf42de-dcd6-4569-9eb9-3e3a53c101f0\") " pod="openshift-network-operator/iptables-alerter-6hkgq" Jul 07 14:34:29.629221 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.628703 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-run-openvswitch\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.629221 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.628760 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8945047-a8e7-4958-9133-a5af6e9ac2b1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.629221 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.628771 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ded5607-54b2-4314-bb3f-443bed8150e6-konnectivity-ca\") pod \"konnectivity-agent-2rp28\" (UID: \"2ded5607-54b2-4314-bb3f-443bed8150e6\") " pod="kube-system/konnectivity-agent-2rp28" Jul 07 14:34:29.629221 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.628820 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-lib-modules\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.629221 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.628856 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95lfm\" (UniqueName: \"kubernetes.io/projected/7fcf42de-dcd6-4569-9eb9-3e3a53c101f0-kube-api-access-95lfm\") pod \"iptables-alerter-6hkgq\" (UID: \"7fcf42de-dcd6-4569-9eb9-3e3a53c101f0\") " pod="openshift-network-operator/iptables-alerter-6hkgq" Jul 07 14:34:29.629221 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.628884 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-cnibin\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.629221 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.628919 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a0f179d-66e3-41bd-aa10-71efe8001185-dbus\") pod \"global-pull-secret-syncer-sjd5c\" (UID: \"3a0f179d-66e3-41bd-aa10-71efe8001185\") " pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:29.629221 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.628951 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b602982-27d1-4239-b9c7-7e09013cf3d5-cni-binary-copy\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.629221 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.628988 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-host-var-lib-cni-bin\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.629221 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.629023 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-cni-netd\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.629221 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.629049 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4jm8\" (UniqueName: \"kubernetes.io/projected/e8c3c836-a810-43f4-bed4-c66cc24e365f-kube-api-access-t4jm8\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.630040 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.629084 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80308b12-3ccc-477c-9578-1ac78d11db32-serviceca\") pod \"node-ca-h78mx\" (UID: \"80308b12-3ccc-477c-9578-1ac78d11db32\") " pod="openshift-image-registry/node-ca-h78mx" Jul 07 14:34:29.630040 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.629119 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ded5607-54b2-4314-bb3f-443bed8150e6-agent-certs\") pod \"konnectivity-agent-2rp28\" (UID: \"2ded5607-54b2-4314-bb3f-443bed8150e6\") " pod="kube-system/konnectivity-agent-2rp28" Jul 07 14:34:29.630040 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.629153 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-etc-openvswitch\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.630040 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.629181 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-log-socket\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.630040 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.629218 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-kubernetes\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.630040 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.629231 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8945047-a8e7-4958-9133-a5af6e9ac2b1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.630040 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.629370 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-lib-modules\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.630040 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.629427 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-host-cni-netd\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.630040 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.629684 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-host-var-lib-cni-bin\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.630040 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.629743 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b602982-27d1-4239-b9c7-7e09013cf3d5-cnibin\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.630040 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.629808 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a0f179d-66e3-41bd-aa10-71efe8001185-dbus\") pod \"global-pull-secret-syncer-sjd5c\" (UID: \"3a0f179d-66e3-41bd-aa10-71efe8001185\") " pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:29.630698 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.630203 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80308b12-3ccc-477c-9578-1ac78d11db32-serviceca\") pod \"node-ca-h78mx\" (UID: \"80308b12-3ccc-477c-9578-1ac78d11db32\") " pod="openshift-image-registry/node-ca-h78mx" Jul 07 14:34:29.630698 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.630296 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-tuned\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.630698 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.630477 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8883f2e-855d-4c69-be5e-2a09c3e9686d-tmp\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.630698 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.630574 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8883f2e-855d-4c69-be5e-2a09c3e9686d-etc-kubernetes\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.630698 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.630625 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-log-socket\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.630941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.630717 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b602982-27d1-4239-b9c7-7e09013cf3d5-cni-binary-copy\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.631021 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.630991 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e8c3c836-a810-43f4-bed4-c66cc24e365f-env-overrides\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.631089 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.631037 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8c3c836-a810-43f4-bed4-c66cc24e365f-ovn-node-metrics-cert\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.632452 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.632347 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8c3c836-a810-43f4-bed4-c66cc24e365f-etc-openvswitch\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.632832 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.632811 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ded5607-54b2-4314-bb3f-443bed8150e6-agent-certs\") pod \"konnectivity-agent-2rp28\" (UID: \"2ded5607-54b2-4314-bb3f-443bed8150e6\") " pod="kube-system/konnectivity-agent-2rp28" Jul 07 14:34:29.634345 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:29.634320 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:34:29.634345 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:29.634349 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:34:29.634559 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:29.634367 2583 projected.go:196] Error preparing data for projected volume kube-api-access-4pptp for pod openshift-network-diagnostics/network-check-target-8d4jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:29.634559 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:29.634521 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp podName:b7a37935-544f-4f5d-8392-2ef44c031582 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:30.134502547 +0000 UTC m=+3.045696236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4pptp" (UniqueName: "kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp") pod "network-check-target-8d4jp" (UID: "b7a37935-544f-4f5d-8392-2ef44c031582") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:29.636740 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.636714 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjwxh\" (UniqueName: \"kubernetes.io/projected/a8945047-a8e7-4958-9133-a5af6e9ac2b1-kube-api-access-zjwxh\") pod \"multus-additional-cni-plugins-pn6lw\" (UID: \"a8945047-a8e7-4958-9133-a5af6e9ac2b1\") " pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.638076 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.638049 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqdjc\" (UniqueName: \"kubernetes.io/projected/c8883f2e-855d-4c69-be5e-2a09c3e9686d-kube-api-access-dqdjc\") pod \"tuned-b8dkc\" (UID: \"c8883f2e-855d-4c69-be5e-2a09c3e9686d\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.638602 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.638577 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kprh\" (UniqueName: \"kubernetes.io/projected/80308b12-3ccc-477c-9578-1ac78d11db32-kube-api-access-2kprh\") pod \"node-ca-h78mx\" (UID: \"80308b12-3ccc-477c-9578-1ac78d11db32\") " pod="openshift-image-registry/node-ca-h78mx" Jul 07 14:34:29.638704 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.638595 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cltf\" (UniqueName: \"kubernetes.io/projected/4b602982-27d1-4239-b9c7-7e09013cf3d5-kube-api-access-9cltf\") pod \"multus-jd2vc\" (UID: \"4b602982-27d1-4239-b9c7-7e09013cf3d5\") " pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.639575 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.639246 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5hbv\" (UniqueName: \"kubernetes.io/projected/1f877ac1-ec8e-437e-a186-28b499cf63e1-kube-api-access-k5hbv\") pod \"node-resolver-rzkvv\" (UID: \"1f877ac1-ec8e-437e-a186-28b499cf63e1\") " pod="openshift-dns/node-resolver-rzkvv" Jul 07 14:34:29.639575 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.639350 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8qjv\" (UniqueName: \"kubernetes.io/projected/ffb5c2bb-7622-42bc-990b-e5441e71ba0b-kube-api-access-q8qjv\") pod \"aws-ebs-csi-driver-node-xll9n\" (UID: \"ffb5c2bb-7622-42bc-990b-e5441e71ba0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.640475 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.640453 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95lfm\" (UniqueName: \"kubernetes.io/projected/7fcf42de-dcd6-4569-9eb9-3e3a53c101f0-kube-api-access-95lfm\") pod \"iptables-alerter-6hkgq\" (UID: \"7fcf42de-dcd6-4569-9eb9-3e3a53c101f0\") " pod="openshift-network-operator/iptables-alerter-6hkgq" Jul 07 14:34:29.640636 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.640619 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4jm8\" (UniqueName: \"kubernetes.io/projected/e8c3c836-a810-43f4-bed4-c66cc24e365f-kube-api-access-t4jm8\") pod \"ovnkube-node-5r6r4\" (UID: \"e8c3c836-a810-43f4-bed4-c66cc24e365f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.643556 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.643532 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwlvv\" (UniqueName: \"kubernetes.io/projected/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-kube-api-access-jwlvv\") pod \"network-metrics-daemon-xsfqs\" (UID: \"d60cabb9-29c9-4f9d-b147-2ca1844f5d96\") " pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:29.790559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.790464 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal" event={"ID":"2f554e2b8de4942207b69bb5df078360","Type":"ContainerStarted","Data":"54f1e9f8e823e403587913249acb7785a4686a34dc8a2dee8d1846c49b360ebe"} Jul 07 14:34:29.791610 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.791582 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:29.791897 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.791860 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" event={"ID":"b14e6cdbc81539fdf89e4bd7f5a99a19","Type":"ContainerStarted","Data":"86e9e103f53244cce26e7c3755273d1b87952ab124f3a85767dfca9523f3115d"} Jul 07 14:34:29.798276 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.798255 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pn6lw" Jul 07 14:34:29.803805 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:29.803776 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c3c836_a810_43f4_bed4_c66cc24e365f.slice/crio-978bad317325ebfb3d3006fcb56f92f604ba6940922b1bdb615bb13f5ded03e1 WatchSource:0}: Error finding container 978bad317325ebfb3d3006fcb56f92f604ba6940922b1bdb615bb13f5ded03e1: Status 404 returned error can't find the container with id 978bad317325ebfb3d3006fcb56f92f604ba6940922b1bdb615bb13f5ded03e1 Jul 07 14:34:29.807704 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.807681 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" Jul 07 14:34:29.809557 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:29.809529 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8945047_a8e7_4958_9133_a5af6e9ac2b1.slice/crio-495965a126b66f3147f74288fcc7f14b4eeba941852d18a76818f7c99f4e5cdb WatchSource:0}: Error finding container 495965a126b66f3147f74288fcc7f14b4eeba941852d18a76818f7c99f4e5cdb: Status 404 returned error can't find the container with id 495965a126b66f3147f74288fcc7f14b4eeba941852d18a76818f7c99f4e5cdb Jul 07 14:34:29.812229 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.812129 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6hkgq" Jul 07 14:34:29.818300 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.818234 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" Jul 07 14:34:29.818568 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:29.818541 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8883f2e_855d_4c69_be5e_2a09c3e9686d.slice/crio-15acd416d25d2e4d6c8acb0c37f81ad6bb27527b6ee3fe10b0c960fe67db1a77 WatchSource:0}: Error finding container 15acd416d25d2e4d6c8acb0c37f81ad6bb27527b6ee3fe10b0c960fe67db1a77: Status 404 returned error can't find the container with id 15acd416d25d2e4d6c8acb0c37f81ad6bb27527b6ee3fe10b0c960fe67db1a77 Jul 07 14:34:29.821651 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:29.821624 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcf42de_dcd6_4569_9eb9_3e3a53c101f0.slice/crio-399696b851bf153c780e46762435f37b62103efd426c2ea9e77b6a960aac1db7 WatchSource:0}: Error finding container 399696b851bf153c780e46762435f37b62103efd426c2ea9e77b6a960aac1db7: Status 404 returned error can't find the container with id 399696b851bf153c780e46762435f37b62103efd426c2ea9e77b6a960aac1db7 Jul 07 14:34:29.825593 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.825554 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jd2vc" Jul 07 14:34:29.827869 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:29.827844 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb5c2bb_7622_42bc_990b_e5441e71ba0b.slice/crio-994e9b1bfe223c94281eb11f86ce1c0f5c512429063aa1aa75e14176c377c8fb WatchSource:0}: Error finding container 994e9b1bfe223c94281eb11f86ce1c0f5c512429063aa1aa75e14176c377c8fb: Status 404 returned error can't find the container with id 994e9b1bfe223c94281eb11f86ce1c0f5c512429063aa1aa75e14176c377c8fb Jul 07 14:34:29.831049 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.831026 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rzkvv" Jul 07 14:34:29.835984 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:29.835925 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b602982_27d1_4239_b9c7_7e09013cf3d5.slice/crio-3a505d1e3cb08459674e951256bc2d2b4d90e12592953927f8b7d48699dffeb2 WatchSource:0}: Error finding container 3a505d1e3cb08459674e951256bc2d2b4d90e12592953927f8b7d48699dffeb2: Status 404 returned error can't find the container with id 3a505d1e3cb08459674e951256bc2d2b4d90e12592953927f8b7d48699dffeb2 Jul 07 14:34:29.836622 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.836594 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h78mx" Jul 07 14:34:29.840526 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:29.840508 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f877ac1_ec8e_437e_a186_28b499cf63e1.slice/crio-30da713d15480db3a181e3ae075a4ab6661a1ffc9d3f64a2cdfaa48e175f6c52 WatchSource:0}: Error finding container 30da713d15480db3a181e3ae075a4ab6661a1ffc9d3f64a2cdfaa48e175f6c52: Status 404 returned error can't find the container with id 30da713d15480db3a181e3ae075a4ab6661a1ffc9d3f64a2cdfaa48e175f6c52 Jul 07 14:34:29.841354 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.841337 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2rp28" Jul 07 14:34:29.846428 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:29.846386 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80308b12_3ccc_477c_9578_1ac78d11db32.slice/crio-21e41be0491d66f9a808859cc4e537e9f56c72bf253074ae0d435726d9cf904d WatchSource:0}: Error finding container 21e41be0491d66f9a808859cc4e537e9f56c72bf253074ae0d435726d9cf904d: Status 404 returned error can't find the container with id 21e41be0491d66f9a808859cc4e537e9f56c72bf253074ae0d435726d9cf904d Jul 07 14:34:29.851317 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:34:29.851288 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ded5607_54b2_4314_bb3f_443bed8150e6.slice/crio-21b6c49d54d9ff77c33a83340bcbd995d0bb2cf0a466162c8293eb27ef888a16 WatchSource:0}: Error finding container 21b6c49d54d9ff77c33a83340bcbd995d0bb2cf0a466162c8293eb27ef888a16: Status 404 returned error can't find the container with id 21b6c49d54d9ff77c33a83340bcbd995d0bb2cf0a466162c8293eb27ef888a16 Jul 07 14:34:29.937888 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:29.937865 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:34:30.133271 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:30.133198 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret\") pod \"global-pull-secret-syncer-sjd5c\" (UID: \"3a0f179d-66e3-41bd-aa10-71efe8001185\") " pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:30.133271 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:30.133237 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs\") pod \"network-metrics-daemon-xsfqs\" (UID: \"d60cabb9-29c9-4f9d-b147-2ca1844f5d96\") " pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:30.133508 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:30.133396 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:30.133508 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:30.133405 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:30.133508 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:30.133476 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs podName:d60cabb9-29c9-4f9d-b147-2ca1844f5d96 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:31.133454246 +0000 UTC m=+4.044647905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs") pod "network-metrics-daemon-xsfqs" (UID: "d60cabb9-29c9-4f9d-b147-2ca1844f5d96") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:30.133660 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:30.133500 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret podName:3a0f179d-66e3-41bd-aa10-71efe8001185 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:31.133488582 +0000 UTC m=+4.044682244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret") pod "global-pull-secret-syncer-sjd5c" (UID: "3a0f179d-66e3-41bd-aa10-71efe8001185") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:30.234373 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:30.234325 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pptp\" (UniqueName: \"kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp\") pod \"network-check-target-8d4jp\" (UID: \"b7a37935-544f-4f5d-8392-2ef44c031582\") " pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:30.234522 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:30.234483 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:34:30.234522 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:30.234504 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:34:30.234522 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:30.234516 2583 projected.go:196] Error preparing data for projected volume kube-api-access-4pptp for pod openshift-network-diagnostics/network-check-target-8d4jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:30.234696 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:30.234576 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp podName:b7a37935-544f-4f5d-8392-2ef44c031582 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:31.234557548 +0000 UTC m=+4.145751220 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4pptp" (UniqueName: "kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp") pod "network-check-target-8d4jp" (UID: "b7a37935-544f-4f5d-8392-2ef44c031582") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:30.311108 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:30.311076 2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:34:30.556478 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:30.556343 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:29:29 +0000 UTC" deadline="2028-03-31 21:05:34.71316731 +0000 UTC" Jul 07 14:34:30.556478 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:30.556417 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15198h31m4.156754749s" Jul 07 14:34:30.787488 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:30.786643 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:30.787488 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:30.786774 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjd5c" podUID="3a0f179d-66e3-41bd-aa10-71efe8001185" Jul 07 14:34:30.787488 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:30.787182 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:30.787488 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:30.787300 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:34:30.787488 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:30.787364 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:30.788509 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:30.788081 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d4jp" podUID="b7a37935-544f-4f5d-8392-2ef44c031582" Jul 07 14:34:30.804844 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:30.804729 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pn6lw" event={"ID":"a8945047-a8e7-4958-9133-a5af6e9ac2b1","Type":"ContainerStarted","Data":"495965a126b66f3147f74288fcc7f14b4eeba941852d18a76818f7c99f4e5cdb"} Jul 07 14:34:30.811311 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:30.811204 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6hkgq" event={"ID":"7fcf42de-dcd6-4569-9eb9-3e3a53c101f0","Type":"ContainerStarted","Data":"399696b851bf153c780e46762435f37b62103efd426c2ea9e77b6a960aac1db7"} Jul 07 14:34:30.814995 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:30.814969 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" event={"ID":"e8c3c836-a810-43f4-bed4-c66cc24e365f","Type":"ContainerStarted","Data":"978bad317325ebfb3d3006fcb56f92f604ba6940922b1bdb615bb13f5ded03e1"} Jul 07 14:34:30.818329 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:30.818259 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2rp28" event={"ID":"2ded5607-54b2-4314-bb3f-443bed8150e6","Type":"ContainerStarted","Data":"21b6c49d54d9ff77c33a83340bcbd995d0bb2cf0a466162c8293eb27ef888a16"} Jul 07 14:34:30.825506 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:30.825430 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h78mx" event={"ID":"80308b12-3ccc-477c-9578-1ac78d11db32","Type":"ContainerStarted","Data":"21e41be0491d66f9a808859cc4e537e9f56c72bf253074ae0d435726d9cf904d"} Jul 07 14:34:30.829068 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:30.828910 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rzkvv" event={"ID":"1f877ac1-ec8e-437e-a186-28b499cf63e1","Type":"ContainerStarted","Data":"30da713d15480db3a181e3ae075a4ab6661a1ffc9d3f64a2cdfaa48e175f6c52"} Jul 07 14:34:30.834897 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:30.834873 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jd2vc" event={"ID":"4b602982-27d1-4239-b9c7-7e09013cf3d5","Type":"ContainerStarted","Data":"3a505d1e3cb08459674e951256bc2d2b4d90e12592953927f8b7d48699dffeb2"} Jul 07 14:34:30.841873 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:30.841849 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" event={"ID":"ffb5c2bb-7622-42bc-990b-e5441e71ba0b","Type":"ContainerStarted","Data":"994e9b1bfe223c94281eb11f86ce1c0f5c512429063aa1aa75e14176c377c8fb"} Jul 07 14:34:30.848190 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:30.848168 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" event={"ID":"c8883f2e-855d-4c69-be5e-2a09c3e9686d","Type":"ContainerStarted","Data":"15acd416d25d2e4d6c8acb0c37f81ad6bb27527b6ee3fe10b0c960fe67db1a77"} Jul 07 14:34:31.142143 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:31.142025 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret\") pod \"global-pull-secret-syncer-sjd5c\" (UID: \"3a0f179d-66e3-41bd-aa10-71efe8001185\") " pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:31.142143 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:31.142124 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs\") pod \"network-metrics-daemon-xsfqs\" (UID: \"d60cabb9-29c9-4f9d-b147-2ca1844f5d96\") " pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:31.142358 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:31.142197 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:31.142358 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:31.142274 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret podName:3a0f179d-66e3-41bd-aa10-71efe8001185 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:33.142251669 +0000 UTC m=+6.053445341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret") pod "global-pull-secret-syncer-sjd5c" (UID: "3a0f179d-66e3-41bd-aa10-71efe8001185") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:31.142358 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:31.142307 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:31.142554 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:31.142363 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs podName:d60cabb9-29c9-4f9d-b147-2ca1844f5d96 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:33.142344974 +0000 UTC m=+6.053538632 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs") pod "network-metrics-daemon-xsfqs" (UID: "d60cabb9-29c9-4f9d-b147-2ca1844f5d96") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:31.243233 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:31.243200 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pptp\" (UniqueName: \"kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp\") pod \"network-check-target-8d4jp\" (UID: \"b7a37935-544f-4f5d-8392-2ef44c031582\") " pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:31.243398 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:31.243378 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:34:31.243484 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:31.243416 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:34:31.243484 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:31.243430 2583 projected.go:196] Error preparing data for projected volume kube-api-access-4pptp for pod openshift-network-diagnostics/network-check-target-8d4jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:31.243595 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:31.243538 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp podName:b7a37935-544f-4f5d-8392-2ef44c031582 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:33.243505408 +0000 UTC m=+6.154699079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4pptp" (UniqueName: "kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp") pod "network-check-target-8d4jp" (UID: "b7a37935-544f-4f5d-8392-2ef44c031582") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:31.858125 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:31.858085 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal" event={"ID":"2f554e2b8de4942207b69bb5df078360","Type":"ContainerStarted","Data":"5b647099de8913f340a6d69752a15e34e0632161b09877b5532a344fcdd4e032"} Jul 07 14:34:32.787029 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:32.786980 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:32.787358 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:32.787118 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjd5c" podUID="3a0f179d-66e3-41bd-aa10-71efe8001185" Jul 07 14:34:32.787647 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:32.787625 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:32.787762 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:32.787741 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:34:32.787871 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:32.787844 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:32.788036 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:32.788010 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d4jp" podUID="b7a37935-544f-4f5d-8392-2ef44c031582" Jul 07 14:34:33.164937 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:33.164841 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret\") pod \"global-pull-secret-syncer-sjd5c\" (UID: \"3a0f179d-66e3-41bd-aa10-71efe8001185\") " pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:33.164937 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:33.164898 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs\") pod \"network-metrics-daemon-xsfqs\" (UID: \"d60cabb9-29c9-4f9d-b147-2ca1844f5d96\") " pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:33.165517 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:33.165066 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:33.165517 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:33.165135 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs podName:d60cabb9-29c9-4f9d-b147-2ca1844f5d96 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:37.165115175 +0000 UTC m=+10.076308840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs") pod "network-metrics-daemon-xsfqs" (UID: "d60cabb9-29c9-4f9d-b147-2ca1844f5d96") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:33.165517 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:33.165206 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:33.165517 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:33.165244 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret podName:3a0f179d-66e3-41bd-aa10-71efe8001185 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:37.165231226 +0000 UTC m=+10.076424884 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret") pod "global-pull-secret-syncer-sjd5c" (UID: "3a0f179d-66e3-41bd-aa10-71efe8001185") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:33.266947 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:33.266229 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pptp\" (UniqueName: \"kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp\") pod \"network-check-target-8d4jp\" (UID: \"b7a37935-544f-4f5d-8392-2ef44c031582\") " pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:33.266947 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:33.266465 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:34:33.266947 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:33.266490 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:34:33.266947 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:33.266505 2583 projected.go:196] Error preparing data for projected volume kube-api-access-4pptp for pod openshift-network-diagnostics/network-check-target-8d4jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:33.266947 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:33.266565 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp podName:b7a37935-544f-4f5d-8392-2ef44c031582 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:37.266545517 +0000 UTC m=+10.177739180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4pptp" (UniqueName: "kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp") pod "network-check-target-8d4jp" (UID: "b7a37935-544f-4f5d-8392-2ef44c031582") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:34.786666 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:34.786609 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:34.787177 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:34.786680 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:34.787177 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:34.786791 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d4jp" podUID="b7a37935-544f-4f5d-8392-2ef44c031582" Jul 07 14:34:34.787177 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:34.786837 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:34.787177 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:34.786923 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjd5c" podUID="3a0f179d-66e3-41bd-aa10-71efe8001185" Jul 07 14:34:34.787549 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:34.787427 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:34:36.787143 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:36.786595 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:36.787143 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:36.786721 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d4jp" podUID="b7a37935-544f-4f5d-8392-2ef44c031582" Jul 07 14:34:36.787143 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:36.787121 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:36.787710 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:36.787223 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:34:36.787710 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:36.787123 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:36.787710 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:36.787318 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjd5c" podUID="3a0f179d-66e3-41bd-aa10-71efe8001185" Jul 07 14:34:37.198022 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:37.197926 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret\") pod \"global-pull-secret-syncer-sjd5c\" (UID: \"3a0f179d-66e3-41bd-aa10-71efe8001185\") " pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:37.198022 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:37.197974 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs\") pod \"network-metrics-daemon-xsfqs\" (UID: \"d60cabb9-29c9-4f9d-b147-2ca1844f5d96\") " pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:37.198234 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:37.198097 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:37.198234 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:37.198125 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:37.198234 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:37.198149 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs podName:d60cabb9-29c9-4f9d-b147-2ca1844f5d96 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:45.198131915 +0000 UTC m=+18.109325584 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs") pod "network-metrics-daemon-xsfqs" (UID: "d60cabb9-29c9-4f9d-b147-2ca1844f5d96") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:37.198234 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:37.198173 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret podName:3a0f179d-66e3-41bd-aa10-71efe8001185 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:45.198161662 +0000 UTC m=+18.109355323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret") pod "global-pull-secret-syncer-sjd5c" (UID: "3a0f179d-66e3-41bd-aa10-71efe8001185") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:37.298962 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:37.298758 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pptp\" (UniqueName: \"kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp\") pod \"network-check-target-8d4jp\" (UID: \"b7a37935-544f-4f5d-8392-2ef44c031582\") " pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:37.298962 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:37.298948 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:34:37.298962 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:37.298966 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:34:37.299242 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:37.298978 2583 projected.go:196] Error preparing data for projected volume kube-api-access-4pptp for pod openshift-network-diagnostics/network-check-target-8d4jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:37.299242 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:37.299032 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp podName:b7a37935-544f-4f5d-8392-2ef44c031582 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:45.299014208 +0000 UTC m=+18.210207895 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4pptp" (UniqueName: "kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp") pod "network-check-target-8d4jp" (UID: "b7a37935-544f-4f5d-8392-2ef44c031582") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:38.786585 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:38.786553 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:38.786585 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:38.786580 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:38.787123 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:38.786687 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d4jp" podUID="b7a37935-544f-4f5d-8392-2ef44c031582" Jul 07 14:34:38.787123 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:38.786699 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:38.787123 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:38.786770 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjd5c" podUID="3a0f179d-66e3-41bd-aa10-71efe8001185" Jul 07 14:34:38.787123 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:38.786819 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:34:40.786336 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:40.786303 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:40.786789 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:40.786303 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:40.786789 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:40.786303 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:40.786789 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:40.786455 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:34:40.786789 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:40.786502 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d4jp" podUID="b7a37935-544f-4f5d-8392-2ef44c031582" Jul 07 14:34:40.786789 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:40.786603 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjd5c" podUID="3a0f179d-66e3-41bd-aa10-71efe8001185" Jul 07 14:34:42.786889 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:42.786858 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:42.787326 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:42.786857 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:42.787326 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:42.786958 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjd5c" podUID="3a0f179d-66e3-41bd-aa10-71efe8001185" Jul 07 14:34:42.787326 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:42.786858 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:42.787326 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:42.787039 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:34:42.787326 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:42.787097 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d4jp" podUID="b7a37935-544f-4f5d-8392-2ef44c031582" Jul 07 14:34:44.786612 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:44.786579 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:44.787116 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:44.786693 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:44.787116 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:44.786705 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjd5c" podUID="3a0f179d-66e3-41bd-aa10-71efe8001185" Jul 07 14:34:44.787116 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:44.786730 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:44.787116 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:44.786802 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d4jp" podUID="b7a37935-544f-4f5d-8392-2ef44c031582" Jul 07 14:34:44.787116 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:44.786889 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:34:45.256012 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:45.255920 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret\") pod \"global-pull-secret-syncer-sjd5c\" (UID: \"3a0f179d-66e3-41bd-aa10-71efe8001185\") " pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:45.256012 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:45.255958 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs\") pod \"network-metrics-daemon-xsfqs\" (UID: \"d60cabb9-29c9-4f9d-b147-2ca1844f5d96\") " pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:45.256279 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:45.256059 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:45.256279 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:45.256061 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:45.256279 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:45.256108 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs podName:d60cabb9-29c9-4f9d-b147-2ca1844f5d96 nodeName:}" failed. No retries permitted until 2026-07-07 14:35:01.256094524 +0000 UTC m=+34.167288180 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs") pod "network-metrics-daemon-xsfqs" (UID: "d60cabb9-29c9-4f9d-b147-2ca1844f5d96") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:34:45.256279 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:45.256121 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret podName:3a0f179d-66e3-41bd-aa10-71efe8001185 nodeName:}" failed. No retries permitted until 2026-07-07 14:35:01.25611574 +0000 UTC m=+34.167309395 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret") pod "global-pull-secret-syncer-sjd5c" (UID: "3a0f179d-66e3-41bd-aa10-71efe8001185") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:34:45.356655 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:45.356623 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pptp\" (UniqueName: \"kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp\") pod \"network-check-target-8d4jp\" (UID: \"b7a37935-544f-4f5d-8392-2ef44c031582\") " pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:45.356788 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:45.356775 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:34:45.356852 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:45.356800 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:34:45.356852 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:45.356812 2583 projected.go:196] Error preparing data for projected volume kube-api-access-4pptp for pod openshift-network-diagnostics/network-check-target-8d4jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:45.356922 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:45.356863 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp podName:b7a37935-544f-4f5d-8392-2ef44c031582 nodeName:}" failed. No retries permitted until 2026-07-07 14:35:01.356848561 +0000 UTC m=+34.268042230 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4pptp" (UniqueName: "kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp") pod "network-check-target-8d4jp" (UID: "b7a37935-544f-4f5d-8392-2ef44c031582") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:34:46.786140 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:46.786109 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:46.786584 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:46.786220 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:46.786584 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:46.786227 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d4jp" podUID="b7a37935-544f-4f5d-8392-2ef44c031582" Jul 07 14:34:46.786584 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:46.786320 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjd5c" podUID="3a0f179d-66e3-41bd-aa10-71efe8001185" Jul 07 14:34:46.786584 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:46.786225 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:46.786584 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:46.786425 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:34:48.786977 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:48.786941 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:48.787408 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:48.787051 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:48.787408 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:48.787053 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjd5c" podUID="3a0f179d-66e3-41bd-aa10-71efe8001185" Jul 07 14:34:48.787408 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:48.787105 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:48.787408 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:48.787185 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d4jp" podUID="b7a37935-544f-4f5d-8392-2ef44c031582" Jul 07 14:34:48.787408 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:48.787269 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:34:49.900802 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:49.900774 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" event={"ID":"c8883f2e-855d-4c69-be5e-2a09c3e9686d","Type":"ContainerStarted","Data":"9eebc8151867903b0b5a8025ae692728f55347c6f73e27e6e29a8a3977308306"} Jul 07 14:34:49.903217 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:49.903195 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" event={"ID":"e8c3c836-a810-43f4-bed4-c66cc24e365f","Type":"ContainerStarted","Data":"864512cbd46a7390232fad980df9f665fc2a6ab6247fbd60fdffa35f4cf84d2b"} Jul 07 14:34:49.903298 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:49.903265 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" event={"ID":"e8c3c836-a810-43f4-bed4-c66cc24e365f","Type":"ContainerStarted","Data":"383726ed807368a27868da87b92e9f2057ecccaa437634b377df29c7183d105d"} Jul 07 14:34:49.903337 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:49.903308 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" event={"ID":"e8c3c836-a810-43f4-bed4-c66cc24e365f","Type":"ContainerStarted","Data":"10701aa0a70285b54a29bab6d487206f86a90fdf8fae12446b8a98887b0a8ccc"} Jul 07 14:34:49.903391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:49.903336 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" event={"ID":"e8c3c836-a810-43f4-bed4-c66cc24e365f","Type":"ContainerStarted","Data":"bd2fb62361c7b990079ec0f04f7b43346af1e1d4555c780d409e81d57bec9cf7"} Jul 07 14:34:49.903435 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:49.903416 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" event={"ID":"e8c3c836-a810-43f4-bed4-c66cc24e365f","Type":"ContainerStarted","Data":"5a0a81e2852a4736cf81861f6e200721a58eaa507e2b88044e7a4035d6514099"} Jul 07 14:34:49.904447 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:49.904428 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jd2vc" event={"ID":"4b602982-27d1-4239-b9c7-7e09013cf3d5","Type":"ContainerStarted","Data":"fd297cd7899220878417908cb593f0e79f3346978c8c89507fc5e0f6d928a581"} Jul 07 14:34:49.939317 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:49.939267 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal" podStartSLOduration=20.939256437 podStartE2EDuration="20.939256437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:31.874854086 +0000 UTC m=+4.786047770" watchObservedRunningTime="2026-07-07 14:34:49.939256437 +0000 UTC m=+22.850450114" Jul 07 14:34:49.940045 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:49.940017 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jd2vc" podStartSLOduration=2.3533779089999998 podStartE2EDuration="21.940008137s" totalImagesPullingTime="19.586630228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:49.938729054 +0000 UTC m=+22.849922734" watchObservedRunningTime="2026-07-07 14:34:49.940008137 +0000 UTC m=+22.851201814" Jul 07 14:34:49.961864 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:49.961828 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b8dkc" podStartSLOduration=2.410529996 podStartE2EDuration="21.961816229s" totalImagesPullingTime="19.551286233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:49.961723608 +0000 UTC m=+22.872917468" watchObservedRunningTime="2026-07-07 14:34:49.961816229 +0000 UTC m=+22.873009889" Jul 07 14:34:50.786669 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:50.786519 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:50.786777 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:50.786581 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:50.786777 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:50.786738 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjd5c" podUID="3a0f179d-66e3-41bd-aa10-71efe8001185" Jul 07 14:34:50.786853 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:50.786822 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:34:50.786853 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:50.786581 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:50.786915 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:50.786900 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d4jp" podUID="b7a37935-544f-4f5d-8392-2ef44c031582" Jul 07 14:34:50.908126 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:50.908057 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2rp28" event={"ID":"2ded5607-54b2-4314-bb3f-443bed8150e6","Type":"ContainerStarted","Data":"3a58072a94d102aa1f0bcace2e3dbdb53f78366cf7f91cf828c44fdb369f84d1"} Jul 07 14:34:50.909727 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:50.909702 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h78mx" event={"ID":"80308b12-3ccc-477c-9578-1ac78d11db32","Type":"ContainerStarted","Data":"62dea05447ac629bc23468302b6d4d77d9687fdc3f354fd99e3e1fb453098567"} Jul 07 14:34:50.911391 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:50.911344 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rzkvv" event={"ID":"1f877ac1-ec8e-437e-a186-28b499cf63e1","Type":"ContainerStarted","Data":"57808a75a9249b99df7ce1150472c2d123f7c2815b77a923cede6d55b4793a2e"} Jul 07 14:34:50.912967 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:50.912946 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" event={"ID":"ffb5c2bb-7622-42bc-990b-e5441e71ba0b","Type":"ContainerStarted","Data":"79e134ddd74578f33f9141717c6216852a9e56d9126da8be14180c2920878ce3"} Jul 07 14:34:50.914473 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:50.914451 2583 generic.go:356] "Generic (PLEG): container finished" podID="a8945047-a8e7-4958-9133-a5af6e9ac2b1" containerID="081bb076e1f5bacef61b8130923b480631b59532d36d10affb9c08ed08787d21" exitCode=0 Jul 07 14:34:50.914569 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:50.914530 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pn6lw" event={"ID":"a8945047-a8e7-4958-9133-a5af6e9ac2b1","Type":"ContainerDied","Data":"081bb076e1f5bacef61b8130923b480631b59532d36d10affb9c08ed08787d21"} Jul 07 14:34:50.916157 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:50.916138 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6hkgq" event={"ID":"7fcf42de-dcd6-4569-9eb9-3e3a53c101f0","Type":"ContainerStarted","Data":"74422776e78c464bf20402b91b99ec151ce4a276c98c0834d52e22debc414ab0"} Jul 07 14:34:50.919293 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:50.919273 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" event={"ID":"e8c3c836-a810-43f4-bed4-c66cc24e365f","Type":"ContainerStarted","Data":"6e1175c71921f2c7c6549bc45981cb77d373c2742bc73e060be45d59191c52c7"} Jul 07 14:34:50.920952 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:50.920929 2583 generic.go:356] "Generic (PLEG): container finished" podID="b14e6cdbc81539fdf89e4bd7f5a99a19" containerID="79482802e5f0c51bf3eb1cd19acd31f812b6660d247f75c60d629d605926e7b4" exitCode=0 Jul 07 14:34:50.921045 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:50.921011 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" event={"ID":"b14e6cdbc81539fdf89e4bd7f5a99a19","Type":"ContainerDied","Data":"79482802e5f0c51bf3eb1cd19acd31f812b6660d247f75c60d629d605926e7b4"} Jul 07 14:34:50.924272 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:50.924214 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2rp28" podStartSLOduration=3.409019193 podStartE2EDuration="22.924196344s" totalImagesPullingTime="19.515177151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:50.923505614 +0000 UTC m=+23.834699293" watchObservedRunningTime="2026-07-07 14:34:50.924196344 +0000 UTC m=+23.835390022" Jul 07 14:34:50.951467 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:50.951430 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6hkgq" podStartSLOduration=3.40526024 podStartE2EDuration="22.951413943s" totalImagesPullingTime="19.546153703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:50.950909324 +0000 UTC m=+23.862102998" watchObservedRunningTime="2026-07-07 14:34:50.951413943 +0000 UTC m=+23.862607621" Jul 07 14:34:50.966047 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:50.966006 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rzkvv" podStartSLOduration=3.438156245 podStartE2EDuration="22.965968471s" totalImagesPullingTime="19.527812226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:50.965512003 +0000 UTC m=+23.876705683" watchObservedRunningTime="2026-07-07 14:34:50.965968471 +0000 UTC m=+23.877162150" Jul 07 14:34:50.980480 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:50.980448 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h78mx" podStartSLOduration=11.454052085 podStartE2EDuration="22.980436737s" totalImagesPullingTime="11.526384652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:50.980139498 +0000 UTC m=+23.891333175" watchObservedRunningTime="2026-07-07 14:34:50.980436737 +0000 UTC m=+23.891630435" Jul 07 14:34:51.544062 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:51.544040 2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 14:34:51.675980 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:51.675905 2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T14:34:51.544060084Z","UUID":"7d1c9f51-e71b-4e61-8bee-342188b09556","Handler":null,"Name":"","Endpoint":""} Jul 07 14:34:51.677177 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:51.677161 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 14:34:51.677268 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:51.677197 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 14:34:51.925082 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:51.925045 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" event={"ID":"b14e6cdbc81539fdf89e4bd7f5a99a19","Type":"ContainerStarted","Data":"ebede3e9bd958576419e7961bf2568730f0a658e5dcfaf701065a89d14484b74"} Jul 07 14:34:51.927019 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:51.926991 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" event={"ID":"ffb5c2bb-7622-42bc-990b-e5441e71ba0b","Type":"ContainerStarted","Data":"783501a6c2ad522c126139cf1ab5841f85ba0a23fd31eb31cbb8b0c9b5230893"} Jul 07 14:34:51.941133 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:51.941097 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" podStartSLOduration=22.941083688 podStartE2EDuration="22.941083688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:51.940927361 +0000 UTC m=+24.852121040" watchObservedRunningTime="2026-07-07 14:34:51.941083688 +0000 UTC m=+24.852277366" Jul 07 14:34:52.786673 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:52.786649 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:52.786804 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:52.786775 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d4jp" podUID="b7a37935-544f-4f5d-8392-2ef44c031582" Jul 07 14:34:52.786804 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:52.786794 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:52.786916 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:52.786781 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:52.786969 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:52.786910 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:34:52.787010 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:52.786981 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjd5c" podUID="3a0f179d-66e3-41bd-aa10-71efe8001185" Jul 07 14:34:52.932094 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:52.932060 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" event={"ID":"ffb5c2bb-7622-42bc-990b-e5441e71ba0b","Type":"ContainerStarted","Data":"96bb5824f2acc2fbb8606994b628bc85d5cc32c867d37ea89011c36793c360c3"} Jul 07 14:34:52.935582 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:52.935557 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" event={"ID":"e8c3c836-a810-43f4-bed4-c66cc24e365f","Type":"ContainerStarted","Data":"3e11ce1473d1f61b0ffda6c7bb0ef45ac71f57768ff30d9ca89a3c9c68d96417"} Jul 07 14:34:52.951196 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:52.951123 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xll9n" podStartSLOduration=3.124581263 podStartE2EDuration="24.951110444s" totalImagesPullingTime="21.826529181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:52.950847157 +0000 UTC m=+25.862040834" watchObservedRunningTime="2026-07-07 14:34:52.951110444 +0000 UTC m=+25.862304123" Jul 07 14:34:54.786772 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:54.786739 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:54.787194 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:54.786739 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:54.787194 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:54.786862 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:34:54.787194 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:54.786752 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:54.787194 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:54.786944 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjd5c" podUID="3a0f179d-66e3-41bd-aa10-71efe8001185" Jul 07 14:34:54.787194 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:54.787002 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d4jp" podUID="b7a37935-544f-4f5d-8392-2ef44c031582" Jul 07 14:34:55.658032 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:55.657831 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2rp28" Jul 07 14:34:55.658393 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:55.658375 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2rp28" Jul 07 14:34:55.944969 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:55.944902 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" event={"ID":"e8c3c836-a810-43f4-bed4-c66cc24e365f","Type":"ContainerStarted","Data":"c774f373078193ce68d64f87fc08abe348778bf3126e637cdf96da14e1d0615d"} Jul 07 14:34:55.945664 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:55.945190 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2rp28" Jul 07 14:34:55.945664 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:55.945237 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:55.945664 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:55.945287 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:55.945664 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:55.945334 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:55.945664 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:55.945611 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2rp28" Jul 07 14:34:55.959618 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:55.959599 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:55.959822 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:55.959805 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:34:55.973162 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:55.973126 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" podStartSLOduration=8.349341702 podStartE2EDuration="27.973113635s" totalImagesPullingTime="19.623771933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:55.972713226 +0000 UTC m=+28.883906904" watchObservedRunningTime="2026-07-07 14:34:55.973113635 +0000 UTC m=+28.884307314" Jul 07 14:34:56.086578 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:56.086558 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:56.179024 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:56.178992 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:56.786900 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:56.786872 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:56.787092 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:56.786965 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d4jp" podUID="b7a37935-544f-4f5d-8392-2ef44c031582" Jul 07 14:34:56.787092 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:56.786971 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:56.787092 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:56.786983 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:56.787092 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:56.787065 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjd5c" podUID="3a0f179d-66e3-41bd-aa10-71efe8001185" Jul 07 14:34:56.787251 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:56.787138 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:34:56.949579 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:56.949539 2583 generic.go:356] "Generic (PLEG): container finished" podID="a8945047-a8e7-4958-9133-a5af6e9ac2b1" containerID="3c34e81058ec82548fec665b6cfd9bf38051f3052db4461855a6f6941b2e93c5" exitCode=0 Jul 07 14:34:56.950182 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:56.949625 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pn6lw" event={"ID":"a8945047-a8e7-4958-9133-a5af6e9ac2b1","Type":"ContainerDied","Data":"3c34e81058ec82548fec665b6cfd9bf38051f3052db4461855a6f6941b2e93c5"} Jul 07 14:34:57.028024 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:57.027941 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:57.128164 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:57.128094 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:57.553792 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:57.553626 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:57.653860 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:57.653835 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:57.948857 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:57.948784 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sjd5c"] Jul 07 14:34:57.948996 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:57.948869 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:57.948996 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:57.948950 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjd5c" podUID="3a0f179d-66e3-41bd-aa10-71efe8001185" Jul 07 14:34:57.952621 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:57.952599 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8d4jp"] Jul 07 14:34:57.952954 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:57.952662 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:57.952954 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:57.952726 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d4jp" podUID="b7a37935-544f-4f5d-8392-2ef44c031582" Jul 07 14:34:57.954225 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:57.954206 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xsfqs"] Jul 07 14:34:57.954298 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:57.954270 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:57.954348 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:57.954333 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:34:58.428073 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:58.428045 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:58.528993 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:58.528960 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:59.181735 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:59.181681 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:59.789616 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:59.789587 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:34:59.789757 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:59.789668 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:34:59.789757 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:59.789670 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjd5c" podUID="3a0f179d-66e3-41bd-aa10-71efe8001185" Jul 07 14:34:59.789757 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:59.789692 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:34:59.789873 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:59.789770 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d4jp" podUID="b7a37935-544f-4f5d-8392-2ef44c031582" Jul 07 14:34:59.789873 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:34:59.789824 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:34:59.957609 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:59.957583 2583 generic.go:356] "Generic (PLEG): container finished" podID="a8945047-a8e7-4958-9133-a5af6e9ac2b1" containerID="5b81f03fd745c0bd4c0619701eea669936f7254d4f48c42e06a1c1d482a80a14" exitCode=0 Jul 07 14:34:59.957708 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:34:59.957618 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pn6lw" event={"ID":"a8945047-a8e7-4958-9133-a5af6e9ac2b1","Type":"ContainerDied","Data":"5b81f03fd745c0bd4c0619701eea669936f7254d4f48c42e06a1c1d482a80a14"} Jul 07 14:35:00.961595 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:00.961383 2583 generic.go:356] "Generic (PLEG): container finished" podID="a8945047-a8e7-4958-9133-a5af6e9ac2b1" containerID="6d8718601b4fbc7b99cce75a8708e3ab16da9d364f23b7c257df28c31c847d11" exitCode=0 Jul 07 14:35:00.961595 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:00.961448 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pn6lw" event={"ID":"a8945047-a8e7-4958-9133-a5af6e9ac2b1","Type":"ContainerDied","Data":"6d8718601b4fbc7b99cce75a8708e3ab16da9d364f23b7c257df28c31c847d11"} Jul 07 14:35:01.274914 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:01.274888 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret\") pod \"global-pull-secret-syncer-sjd5c\" (UID: \"3a0f179d-66e3-41bd-aa10-71efe8001185\") " pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:35:01.275071 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:01.274921 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs\") pod \"network-metrics-daemon-xsfqs\" (UID: \"d60cabb9-29c9-4f9d-b147-2ca1844f5d96\") " pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:35:01.275071 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:01.275016 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:35:01.275071 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:01.275067 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret podName:3a0f179d-66e3-41bd-aa10-71efe8001185 nodeName:}" failed. No retries permitted until 2026-07-07 14:35:33.27505368 +0000 UTC m=+66.186247340 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret") pod "global-pull-secret-syncer-sjd5c" (UID: "3a0f179d-66e3-41bd-aa10-71efe8001185") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:35:01.275176 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:01.275016 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:35:01.275176 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:01.275140 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs podName:d60cabb9-29c9-4f9d-b147-2ca1844f5d96 nodeName:}" failed. No retries permitted until 2026-07-07 14:35:33.275129496 +0000 UTC m=+66.186323156 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs") pod "network-metrics-daemon-xsfqs" (UID: "d60cabb9-29c9-4f9d-b147-2ca1844f5d96") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:35:01.375950 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:01.375919 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pptp\" (UniqueName: \"kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp\") pod \"network-check-target-8d4jp\" (UID: \"b7a37935-544f-4f5d-8392-2ef44c031582\") " pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:35:01.376078 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:01.376036 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:35:01.376078 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:01.376048 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:35:01.376078 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:01.376056 2583 projected.go:196] Error preparing data for projected volume kube-api-access-4pptp for pod openshift-network-diagnostics/network-check-target-8d4jp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:35:01.376169 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:01.376096 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp podName:b7a37935-544f-4f5d-8392-2ef44c031582 nodeName:}" failed. No retries permitted until 2026-07-07 14:35:33.376085037 +0000 UTC m=+66.287278692 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4pptp" (UniqueName: "kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp") pod "network-check-target-8d4jp" (UID: "b7a37935-544f-4f5d-8392-2ef44c031582") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:35:01.786554 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:01.786527 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:35:01.786554 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:01.786543 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:35:01.786950 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:01.786919 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sjd5c" podUID="3a0f179d-66e3-41bd-aa10-71efe8001185" Jul 07 14:35:01.789532 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:01.787301 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:35:01.789532 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:01.787771 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8d4jp" podUID="b7a37935-544f-4f5d-8392-2ef44c031582" Jul 07 14:35:01.789532 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:01.787801 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:35:03.277871 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.277844 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeReady" Jul 07 14:35:03.278228 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.277958 2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 14:35:03.331253 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.331175 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tdw9f"] Jul 07 14:35:03.333235 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.333214 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tdw9f" Jul 07 14:35:03.335756 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.335734 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7dfs5"] Jul 07 14:35:03.336189 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.336006 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 14:35:03.336477 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.336459 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v5tsw\"" Jul 07 14:35:03.336561 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.336462 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 14:35:03.337803 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.337785 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7dfs5" Jul 07 14:35:03.339811 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.339791 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 14:35:03.339811 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.339803 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 14:35:03.340294 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.340276 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 14:35:03.340760 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.340745 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ntb78\"" Jul 07 14:35:03.348068 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.348049 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tdw9f"] Jul 07 14:35:03.352039 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.352018 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7dfs5"] Jul 07 14:35:03.492425 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.492399 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdnw2\" (UniqueName: \"kubernetes.io/projected/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-kube-api-access-rdnw2\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:35:03.492569 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.492440 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chhxl\" (UniqueName: \"kubernetes.io/projected/288a2a6a-6435-4eab-bb18-ff6cf90ec0be-kube-api-access-chhxl\") pod \"ingress-canary-7dfs5\" (UID: \"288a2a6a-6435-4eab-bb18-ff6cf90ec0be\") " pod="openshift-ingress-canary/ingress-canary-7dfs5" Jul 07 14:35:03.492569 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.492539 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:35:03.492678 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.492573 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-tmp-dir\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:35:03.492678 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.492597 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/288a2a6a-6435-4eab-bb18-ff6cf90ec0be-cert\") pod \"ingress-canary-7dfs5\" (UID: \"288a2a6a-6435-4eab-bb18-ff6cf90ec0be\") " pod="openshift-ingress-canary/ingress-canary-7dfs5" Jul 07 14:35:03.492771 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.492693 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-config-volume\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:35:03.593883 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.593809 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdnw2\" (UniqueName: \"kubernetes.io/projected/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-kube-api-access-rdnw2\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:35:03.593883 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.593842 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chhxl\" (UniqueName: \"kubernetes.io/projected/288a2a6a-6435-4eab-bb18-ff6cf90ec0be-kube-api-access-chhxl\") pod \"ingress-canary-7dfs5\" (UID: \"288a2a6a-6435-4eab-bb18-ff6cf90ec0be\") " pod="openshift-ingress-canary/ingress-canary-7dfs5" Jul 07 14:35:03.594082 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.593895 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:35:03.594082 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.593922 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-tmp-dir\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:35:03.594082 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.593943 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/288a2a6a-6435-4eab-bb18-ff6cf90ec0be-cert\") pod \"ingress-canary-7dfs5\" (UID: \"288a2a6a-6435-4eab-bb18-ff6cf90ec0be\") " pod="openshift-ingress-canary/ingress-canary-7dfs5" Jul 07 14:35:03.594082 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.593983 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-config-volume\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:35:03.594338 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:03.594221 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:35:03.594338 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:03.594291 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls podName:e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b nodeName:}" failed. No retries permitted until 2026-07-07 14:35:04.094263258 +0000 UTC m=+37.005456922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls") pod "dns-default-tdw9f" (UID: "e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b") : secret "dns-default-metrics-tls" not found Jul 07 14:35:03.594469 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.594452 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-tmp-dir\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:35:03.594672 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.594644 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-config-volume\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:35:03.598896 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.598871 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/288a2a6a-6435-4eab-bb18-ff6cf90ec0be-cert\") pod \"ingress-canary-7dfs5\" (UID: \"288a2a6a-6435-4eab-bb18-ff6cf90ec0be\") " pod="openshift-ingress-canary/ingress-canary-7dfs5" Jul 07 14:35:03.603972 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.603928 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdnw2\" (UniqueName: \"kubernetes.io/projected/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-kube-api-access-rdnw2\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:35:03.604211 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.604189 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chhxl\" (UniqueName: \"kubernetes.io/projected/288a2a6a-6435-4eab-bb18-ff6cf90ec0be-kube-api-access-chhxl\") pod \"ingress-canary-7dfs5\" (UID: \"288a2a6a-6435-4eab-bb18-ff6cf90ec0be\") " pod="openshift-ingress-canary/ingress-canary-7dfs5" Jul 07 14:35:03.651138 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.651111 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7dfs5" Jul 07 14:35:03.786461 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.786383 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:35:03.786461 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.786399 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:35:03.786461 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.786430 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:35:03.789487 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.789430 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 14:35:03.789487 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.789455 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 14:35:03.790319 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.790028 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 14:35:03.790319 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.790042 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 14:35:03.790319 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.790078 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dhggt\"" Jul 07 14:35:03.790319 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.790153 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v8hh9\"" Jul 07 14:35:03.796046 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.796026 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7dfs5"] Jul 07 14:35:03.800231 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:35:03.800211 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288a2a6a_6435_4eab_bb18_ff6cf90ec0be.slice/crio-56449c813299403d0e24a3c25c3a4c848258e8734430f0b7d1250e6f78a3dfa3 WatchSource:0}: Error finding container 56449c813299403d0e24a3c25c3a4c848258e8734430f0b7d1250e6f78a3dfa3: Status 404 returned error can't find the container with id 56449c813299403d0e24a3c25c3a4c848258e8734430f0b7d1250e6f78a3dfa3 Jul 07 14:35:03.969152 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:03.969106 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7dfs5" event={"ID":"288a2a6a-6435-4eab-bb18-ff6cf90ec0be","Type":"ContainerStarted","Data":"56449c813299403d0e24a3c25c3a4c848258e8734430f0b7d1250e6f78a3dfa3"} Jul 07 14:35:04.098903 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:04.098868 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:35:04.099071 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:04.098991 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:35:04.099071 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:04.099047 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls podName:e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b nodeName:}" failed. No retries permitted until 2026-07-07 14:35:05.099034157 +0000 UTC m=+38.010227813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls") pod "dns-default-tdw9f" (UID: "e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b") : secret "dns-default-metrics-tls" not found Jul 07 14:35:05.106933 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:05.106894 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:35:05.107666 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:05.107047 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:35:05.107666 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:05.107114 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls podName:e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b nodeName:}" failed. No retries permitted until 2026-07-07 14:35:07.107097535 +0000 UTC m=+40.018291191 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls") pod "dns-default-tdw9f" (UID: "e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b") : secret "dns-default-metrics-tls" not found Jul 07 14:35:05.977409 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:05.977377 2583 generic.go:356] "Generic (PLEG): container finished" podID="a8945047-a8e7-4958-9133-a5af6e9ac2b1" containerID="9306705fdf3da57a716d7850bd0fe6274ad69138817cdaade91863515a84d7bc" exitCode=0 Jul 07 14:35:05.977549 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:05.977423 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pn6lw" event={"ID":"a8945047-a8e7-4958-9133-a5af6e9ac2b1","Type":"ContainerDied","Data":"9306705fdf3da57a716d7850bd0fe6274ad69138817cdaade91863515a84d7bc"} Jul 07 14:35:06.423132 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:06.423045 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:06.487618 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:06.487590 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:06.983705 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:06.983661 2583 generic.go:356] "Generic (PLEG): container finished" podID="a8945047-a8e7-4958-9133-a5af6e9ac2b1" containerID="4b01638dd01346c78edc1a858f5d9eaae36273db515a23dd273763d8b83a4294" exitCode=0 Jul 07 14:35:06.983882 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:06.983733 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pn6lw" event={"ID":"a8945047-a8e7-4958-9133-a5af6e9ac2b1","Type":"ContainerDied","Data":"4b01638dd01346c78edc1a858f5d9eaae36273db515a23dd273763d8b83a4294"} Jul 07 14:35:07.125120 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:07.125085 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:35:07.125312 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:07.125290 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:35:07.125393 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:07.125381 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls podName:e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b nodeName:}" failed. No retries permitted until 2026-07-07 14:35:11.125345425 +0000 UTC m=+44.036539098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls") pod "dns-default-tdw9f" (UID: "e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b") : secret "dns-default-metrics-tls" not found Jul 07 14:35:07.200033 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:07.199992 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:07.263324 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:07.263252 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:07.987138 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:07.987101 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7dfs5" event={"ID":"288a2a6a-6435-4eab-bb18-ff6cf90ec0be","Type":"ContainerStarted","Data":"68ff8e0813ed0760e1cccd1accc599175799492bf8f920ca1757979a03801319"} Jul 07 14:35:07.990649 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:07.990626 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pn6lw" event={"ID":"a8945047-a8e7-4958-9133-a5af6e9ac2b1","Type":"ContainerStarted","Data":"ea641f87abb2586131a486f216b6f1667a291c3384364aa4e7ab83eba7720eb4"} Jul 07 14:35:08.002709 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:08.002671 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7dfs5" podStartSLOduration=1.146843857 podStartE2EDuration="5.002657057s" totalImagesPullingTime="3.8558132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:35:08.001932381 +0000 UTC m=+40.913126059" watchObservedRunningTime="2026-07-07 14:35:08.002657057 +0000 UTC m=+40.913850736" Jul 07 14:35:08.025726 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:08.025311 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pn6lw" podStartSLOduration=8.317548135 podStartE2EDuration="40.025294581s" totalImagesPullingTime="31.707746446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:35:08.024014689 +0000 UTC m=+40.935208367" watchObservedRunningTime="2026-07-07 14:35:08.025294581 +0000 UTC m=+40.936488334" Jul 07 14:35:08.147447 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:08.147397 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:08.212513 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:08.212484 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:11.151031 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:11.150988 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:35:11.151443 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:11.151134 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:35:11.151443 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:11.151192 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls podName:e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b nodeName:}" failed. No retries permitted until 2026-07-07 14:35:19.1511771 +0000 UTC m=+52.062370756 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls") pod "dns-default-tdw9f" (UID: "e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b") : secret "dns-default-metrics-tls" not found Jul 07 14:35:19.051324 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:19.051287 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:19.150102 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:19.150069 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:19.202098 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:19.202077 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:35:19.202206 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:19.202193 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:35:19.202255 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:19.202245 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls podName:e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b nodeName:}" failed. No retries permitted until 2026-07-07 14:35:35.202230991 +0000 UTC m=+68.113424647 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls") pod "dns-default-tdw9f" (UID: "e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b") : secret "dns-default-metrics-tls" not found Jul 07 14:35:20.410266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:20.410234 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:20.471936 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:20.471910 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:27.961237 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:27.961212 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5r6r4" Jul 07 14:35:28.089555 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:28.089525 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:28.149431 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:28.149404 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:28.208263 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:28.208242 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:33.242403 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:33.242334 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:33.293416 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:33.293383 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret\") pod \"global-pull-secret-syncer-sjd5c\" (UID: \"3a0f179d-66e3-41bd-aa10-71efe8001185\") " pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:35:33.293559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:33.293433 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs\") pod \"network-metrics-daemon-xsfqs\" (UID: \"d60cabb9-29c9-4f9d-b147-2ca1844f5d96\") " pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:35:33.295839 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:33.295822 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 14:35:33.296006 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:33.295987 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 14:35:33.303813 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:33.303796 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 14:35:33.303911 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:33.303859 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs podName:d60cabb9-29c9-4f9d-b147-2ca1844f5d96 nodeName:}" failed. No retries permitted until 2026-07-07 14:36:37.303840342 +0000 UTC m=+130.215033998 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs") pod "network-metrics-daemon-xsfqs" (UID: "d60cabb9-29c9-4f9d-b147-2ca1844f5d96") : secret "metrics-daemon-secret" not found Jul 07 14:35:33.306896 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:33.306874 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0f179d-66e3-41bd-aa10-71efe8001185-original-pull-secret\") pod \"global-pull-secret-syncer-sjd5c\" (UID: \"3a0f179d-66e3-41bd-aa10-71efe8001185\") " pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:35:33.394185 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:33.394160 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pptp\" (UniqueName: \"kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp\") pod \"network-check-target-8d4jp\" (UID: \"b7a37935-544f-4f5d-8392-2ef44c031582\") " pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:35:33.396558 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:33.396541 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 14:35:33.406826 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:33.406809 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 14:35:33.417832 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:33.417804 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pptp\" (UniqueName: \"kubernetes.io/projected/b7a37935-544f-4f5d-8392-2ef44c031582-kube-api-access-4pptp\") pod \"network-check-target-8d4jp\" (UID: \"b7a37935-544f-4f5d-8392-2ef44c031582\") " pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:35:33.501458 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:33.501397 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dhggt\"" Jul 07 14:35:33.509745 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:33.509728 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:35:33.514935 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:33.514919 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sjd5c" Jul 07 14:35:33.637350 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:33.637321 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sjd5c"] Jul 07 14:35:33.640245 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:35:33.640222 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0f179d_66e3_41bd_aa10_71efe8001185.slice/crio-4820cf1271bc4329546776b88603771ba660f0f5731482f57ecea6ca7faabbeb WatchSource:0}: Error finding container 4820cf1271bc4329546776b88603771ba660f0f5731482f57ecea6ca7faabbeb: Status 404 returned error can't find the container with id 4820cf1271bc4329546776b88603771ba660f0f5731482f57ecea6ca7faabbeb Jul 07 14:35:33.652685 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:33.652661 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8d4jp"] Jul 07 14:35:33.655346 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:35:33.655326 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a37935_544f_4f5d_8392_2ef44c031582.slice/crio-2d5d2e3cf60cba87e5abf216dfee82cbfab3c1716731285b6c867612364ec551 WatchSource:0}: Error finding container 2d5d2e3cf60cba87e5abf216dfee82cbfab3c1716731285b6c867612364ec551: Status 404 returned error can't find the container with id 2d5d2e3cf60cba87e5abf216dfee82cbfab3c1716731285b6c867612364ec551 Jul 07 14:35:34.053126 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:34.053099 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sjd5c" event={"ID":"3a0f179d-66e3-41bd-aa10-71efe8001185","Type":"ContainerStarted","Data":"4820cf1271bc4329546776b88603771ba660f0f5731482f57ecea6ca7faabbeb"} Jul 07 14:35:34.054178 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:34.054159 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8d4jp" event={"ID":"b7a37935-544f-4f5d-8392-2ef44c031582","Type":"ContainerStarted","Data":"2d5d2e3cf60cba87e5abf216dfee82cbfab3c1716731285b6c867612364ec551"} Jul 07 14:35:35.206712 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:35.206678 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:35:35.207161 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:35.206823 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:35:35.207161 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:35:35.206882 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls podName:e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b nodeName:}" failed. No retries permitted until 2026-07-07 14:36:07.206863329 +0000 UTC m=+100.118056991 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls") pod "dns-default-tdw9f" (UID: "e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b") : secret "dns-default-metrics-tls" not found Jul 07 14:35:39.069647 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:39.069604 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sjd5c" event={"ID":"3a0f179d-66e3-41bd-aa10-71efe8001185","Type":"ContainerStarted","Data":"e3c2558d7322ae6638cb191d663bab8645f2c093bc390098232c33fe58093077"} Jul 07 14:35:39.071132 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:39.071110 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8d4jp" event={"ID":"b7a37935-544f-4f5d-8392-2ef44c031582","Type":"ContainerStarted","Data":"e0d3eabb87db186b5932e14bcea096efc7cbfaad2b8237cd8737aadda9b87a41"} Jul 07 14:35:39.071251 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:39.071239 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:35:39.085945 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:39.085901 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sjd5c" podStartSLOduration=66.432626346 podStartE2EDuration="1m11.085888658s" totalImagesPullingTime="4.653262312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:35:39.084638635 +0000 UTC m=+71.995832339" watchObservedRunningTime="2026-07-07 14:35:39.085888658 +0000 UTC m=+71.997082344" Jul 07 14:35:39.101136 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:39.101096 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8d4jp" podStartSLOduration=66.466305572 podStartE2EDuration="1m11.101084093s" totalImagesPullingTime="4.634778521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:35:39.100610437 +0000 UTC m=+72.011804115" watchObservedRunningTime="2026-07-07 14:35:39.101084093 +0000 UTC m=+72.012277770" Jul 07 14:35:57.432018 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:57.431987 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:57.536532 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:35:57.536493 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:07.210986 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:07.210950 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:36:07.211443 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:36:07.211054 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:36:07.211443 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:36:07.211129 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls podName:e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b nodeName:}" failed. No retries permitted until 2026-07-07 14:37:11.211114252 +0000 UTC m=+164.122307908 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls") pod "dns-default-tdw9f" (UID: "e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b") : secret "dns-default-metrics-tls" not found Jul 07 14:36:10.075807 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:10.075779 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8d4jp" Jul 07 14:36:26.808033 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.807982 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm"] Jul 07 14:36:26.810702 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.810687 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:26.813227 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.813204 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 07 14:36:26.813884 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.813866 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 07 14:36:26.814128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.814110 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 07 14:36:26.814208 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.814116 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 07 14:36:26.814208 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.814158 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 07 14:36:26.814208 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.814179 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 07 14:36:26.814374 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.814241 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 07 14:36:26.822440 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.822421 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm"] Jul 07 14:36:26.841152 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.841133 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/38d3d17e-c69d-4b8c-9c8f-4c1f318400ba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm\" (UID: \"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:26.841267 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.841161 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/38d3d17e-c69d-4b8c-9c8f-4c1f318400ba-ca\") pod \"cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm\" (UID: \"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:26.841267 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.841184 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/38d3d17e-c69d-4b8c-9c8f-4c1f318400ba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm\" (UID: \"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:26.841267 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.841207 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d3d17e-c69d-4b8c-9c8f-4c1f318400ba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm\" (UID: \"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:26.841463 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.841315 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bcp6\" (UniqueName: \"kubernetes.io/projected/38d3d17e-c69d-4b8c-9c8f-4c1f318400ba-kube-api-access-2bcp6\") pod \"cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm\" (UID: \"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:26.841463 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.841399 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/38d3d17e-c69d-4b8c-9c8f-4c1f318400ba-hub\") pod \"cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm\" (UID: \"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:26.942602 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.942564 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/38d3d17e-c69d-4b8c-9c8f-4c1f318400ba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm\" (UID: \"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:26.942602 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.942613 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d3d17e-c69d-4b8c-9c8f-4c1f318400ba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm\" (UID: \"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:26.942817 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.942656 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bcp6\" (UniqueName: \"kubernetes.io/projected/38d3d17e-c69d-4b8c-9c8f-4c1f318400ba-kube-api-access-2bcp6\") pod \"cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm\" (UID: \"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:26.942817 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.942684 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/38d3d17e-c69d-4b8c-9c8f-4c1f318400ba-hub\") pod \"cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm\" (UID: \"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:26.942817 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.942715 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/38d3d17e-c69d-4b8c-9c8f-4c1f318400ba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm\" (UID: \"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:26.942817 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.942740 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/38d3d17e-c69d-4b8c-9c8f-4c1f318400ba-ca\") pod \"cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm\" (UID: \"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:26.943535 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.943504 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d3d17e-c69d-4b8c-9c8f-4c1f318400ba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm\" (UID: \"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:26.945193 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.945161 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/38d3d17e-c69d-4b8c-9c8f-4c1f318400ba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm\" (UID: \"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:26.945193 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.945161 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/38d3d17e-c69d-4b8c-9c8f-4c1f318400ba-ca\") pod \"cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm\" (UID: \"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:26.945345 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.945225 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/38d3d17e-c69d-4b8c-9c8f-4c1f318400ba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm\" (UID: \"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:26.945345 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.945233 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/38d3d17e-c69d-4b8c-9c8f-4c1f318400ba-hub\") pod \"cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm\" (UID: \"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:26.951994 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:26.951975 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bcp6\" (UniqueName: \"kubernetes.io/projected/38d3d17e-c69d-4b8c-9c8f-4c1f318400ba-kube-api-access-2bcp6\") pod \"cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm\" (UID: \"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:27.132173 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:27.132089 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:36:27.249702 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:27.249674 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm"] Jul 07 14:36:27.252261 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:36:27.252236 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38d3d17e_c69d_4b8c_9c8f_4c1f318400ba.slice/crio-521aa83f76ef54ca25b8a800799fa03108e1a23c2501df8e74a7765cde99880e WatchSource:0}: Error finding container 521aa83f76ef54ca25b8a800799fa03108e1a23c2501df8e74a7765cde99880e: Status 404 returned error can't find the container with id 521aa83f76ef54ca25b8a800799fa03108e1a23c2501df8e74a7765cde99880e Jul 07 14:36:28.185554 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:28.185516 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" event={"ID":"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba","Type":"ContainerStarted","Data":"521aa83f76ef54ca25b8a800799fa03108e1a23c2501df8e74a7765cde99880e"} Jul 07 14:36:32.198920 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:32.198884 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" event={"ID":"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba","Type":"ContainerStarted","Data":"1973d543a0ac5c43acd65773e80c16ebd9834af7ad23a133f1a24186dcf79b70"} Jul 07 14:36:32.199285 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:32.198929 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" event={"ID":"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba","Type":"ContainerStarted","Data":"6ff8b4a1203b80def917d077b608c354847c5cc419bed67bea0404b327ef1c9c"} Jul 07 14:36:32.199285 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:32.198942 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" event={"ID":"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba","Type":"ContainerStarted","Data":"ab8ce27f789c575015a60a4ad9fa68b7ef9ea1a35fcbaa6d1782ee917421de63"} Jul 07 14:36:32.220976 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:32.220929 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" podStartSLOduration=1.929446021 podStartE2EDuration="6.220915469s" totalImagesPullingTime="4.291469448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:36:32.220176896 +0000 UTC m=+125.131370577" watchObservedRunningTime="2026-07-07 14:36:32.220915469 +0000 UTC m=+125.132109168" Jul 07 14:36:37.315644 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:36:37.315588 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs\") pod \"network-metrics-daemon-xsfqs\" (UID: \"d60cabb9-29c9-4f9d-b147-2ca1844f5d96\") " pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:36:37.316127 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:36:37.315753 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 14:36:37.316127 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:36:37.315826 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs podName:d60cabb9-29c9-4f9d-b147-2ca1844f5d96 nodeName:}" failed. No retries permitted until 2026-07-07 14:38:39.315811485 +0000 UTC m=+252.227005140 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs") pod "network-metrics-daemon-xsfqs" (UID: "d60cabb9-29c9-4f9d-b147-2ca1844f5d96") : secret "metrics-daemon-secret" not found Jul 07 14:37:06.345752 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:37:06.345711 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tdw9f" podUID="e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b" Jul 07 14:37:06.820442 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:37:06.820408 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:37:07.295799 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:37:07.295769 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tdw9f" Jul 07 14:37:11.226936 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:37:11.226899 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:37:11.227423 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:37:11.227015 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:37:11.227423 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:37:11.227073 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls podName:e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b nodeName:}" failed. No retries permitted until 2026-07-07 14:39:13.227055606 +0000 UTC m=+286.138249267 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls") pod "dns-default-tdw9f" (UID: "e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b") : secret "dns-default-metrics-tls" not found Jul 07 14:37:17.788290 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:37:17.788252 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:37:59.877512 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:37:59.877484 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:34.476757 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.476727 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-x5rws"] Jul 07 14:38:34.478793 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.478778 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-x5rws" Jul 07 14:38:34.485394 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.485376 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 07 14:38:34.486103 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.486084 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 07 14:38:34.486234 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.486195 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:38:34.486293 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.486283 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 07 14:38:34.487220 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.487201 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7fcwm\"" Jul 07 14:38:34.491561 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.491543 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 07 14:38:34.508896 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.508873 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-x5rws"] Jul 07 14:38:34.649502 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.649470 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30-serving-cert\") pod \"console-operator-68b757fb67-x5rws\" (UID: \"6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30\") " pod="openshift-console-operator/console-operator-68b757fb67-x5rws" Jul 07 14:38:34.649502 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.649508 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2x97\" (UniqueName: \"kubernetes.io/projected/6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30-kube-api-access-r2x97\") pod \"console-operator-68b757fb67-x5rws\" (UID: \"6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30\") " pod="openshift-console-operator/console-operator-68b757fb67-x5rws" Jul 07 14:38:34.649633 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.649543 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30-config\") pod \"console-operator-68b757fb67-x5rws\" (UID: \"6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30\") " pod="openshift-console-operator/console-operator-68b757fb67-x5rws" Jul 07 14:38:34.649633 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.649566 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30-trusted-ca\") pod \"console-operator-68b757fb67-x5rws\" (UID: \"6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30\") " pod="openshift-console-operator/console-operator-68b757fb67-x5rws" Jul 07 14:38:34.683817 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.683790 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:34.749991 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.749912 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30-config\") pod \"console-operator-68b757fb67-x5rws\" (UID: \"6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30\") " pod="openshift-console-operator/console-operator-68b757fb67-x5rws" Jul 07 14:38:34.749991 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.749963 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30-trusted-ca\") pod \"console-operator-68b757fb67-x5rws\" (UID: \"6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30\") " pod="openshift-console-operator/console-operator-68b757fb67-x5rws" Jul 07 14:38:34.750140 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.750028 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30-serving-cert\") pod \"console-operator-68b757fb67-x5rws\" (UID: \"6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30\") " pod="openshift-console-operator/console-operator-68b757fb67-x5rws" Jul 07 14:38:34.750140 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.750057 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2x97\" (UniqueName: \"kubernetes.io/projected/6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30-kube-api-access-r2x97\") pod \"console-operator-68b757fb67-x5rws\" (UID: \"6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30\") " pod="openshift-console-operator/console-operator-68b757fb67-x5rws" Jul 07 14:38:34.750641 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.750616 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30-config\") pod \"console-operator-68b757fb67-x5rws\" (UID: \"6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30\") " pod="openshift-console-operator/console-operator-68b757fb67-x5rws" Jul 07 14:38:34.750985 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.750967 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30-trusted-ca\") pod \"console-operator-68b757fb67-x5rws\" (UID: \"6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30\") " pod="openshift-console-operator/console-operator-68b757fb67-x5rws" Jul 07 14:38:34.752468 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.752451 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30-serving-cert\") pod \"console-operator-68b757fb67-x5rws\" (UID: \"6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30\") " pod="openshift-console-operator/console-operator-68b757fb67-x5rws" Jul 07 14:38:34.759199 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.759179 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2x97\" (UniqueName: \"kubernetes.io/projected/6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30-kube-api-access-r2x97\") pod \"console-operator-68b757fb67-x5rws\" (UID: \"6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30\") " pod="openshift-console-operator/console-operator-68b757fb67-x5rws" Jul 07 14:38:34.787320 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.787298 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-x5rws" Jul 07 14:38:34.908659 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:34.908570 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-x5rws"] Jul 07 14:38:34.911191 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:38:34.911165 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c67431a_a8aa_4dcf_89e1_4f6e3c35fa30.slice/crio-73b221d9ff6ab8caff896ae80f148f76a13ab04563b80c8599ef57e34ebbc223 WatchSource:0}: Error finding container 73b221d9ff6ab8caff896ae80f148f76a13ab04563b80c8599ef57e34ebbc223: Status 404 returned error can't find the container with id 73b221d9ff6ab8caff896ae80f148f76a13ab04563b80c8599ef57e34ebbc223 Jul 07 14:38:35.538483 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:35.538445 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-x5rws" event={"ID":"6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30","Type":"ContainerStarted","Data":"73b221d9ff6ab8caff896ae80f148f76a13ab04563b80c8599ef57e34ebbc223"} Jul 07 14:38:37.546801 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:37.546775 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/0.log" Jul 07 14:38:37.547170 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:37.546812 2583 generic.go:356] "Generic (PLEG): container finished" podID="6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30" containerID="f2fbc69e372f51b083ac547bc1e344cd31a27b1db31fc3fc7883700830a36681" exitCode=255 Jul 07 14:38:37.547170 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:37.546839 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-x5rws" event={"ID":"6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30","Type":"ContainerDied","Data":"f2fbc69e372f51b083ac547bc1e344cd31a27b1db31fc3fc7883700830a36681"} Jul 07 14:38:37.547170 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:37.547048 2583 scope.go:117] "RemoveContainer" containerID="f2fbc69e372f51b083ac547bc1e344cd31a27b1db31fc3fc7883700830a36681" Jul 07 14:38:37.609593 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:37.609574 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:38.380228 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.380193 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6944g"] Jul 07 14:38:38.382765 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.382742 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:38.389324 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.389031 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 14:38:38.389324 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.389036 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 14:38:38.389324 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.389063 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s99qs\"" Jul 07 14:38:38.389695 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.389679 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 14:38:38.392573 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.392557 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 14:38:38.409630 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.409607 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6944g"] Jul 07 14:38:38.442982 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.442952 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:38.474159 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.474139 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7de534e8-1ede-4999-9322-2944033cf5be-crio-socket\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:38.474226 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.474171 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7de534e8-1ede-4999-9322-2944033cf5be-data-volume\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:38.474226 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.474190 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:38.474226 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.474209 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7de534e8-1ede-4999-9322-2944033cf5be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:38.474335 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.474276 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7lfk\" (UniqueName: \"kubernetes.io/projected/7de534e8-1ede-4999-9322-2944033cf5be-kube-api-access-v7lfk\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:38.509599 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.509580 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:38.550756 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.550738 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 14:38:38.551165 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.551151 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/0.log" Jul 07 14:38:38.551208 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.551186 2583 generic.go:356] "Generic (PLEG): container finished" podID="6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30" containerID="81c411940b0188bdb3c007ab402e4a36f72b62ecf7cda1add5b360837d0b3149" exitCode=255 Jul 07 14:38:38.551241 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.551225 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-x5rws" event={"ID":"6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30","Type":"ContainerDied","Data":"81c411940b0188bdb3c007ab402e4a36f72b62ecf7cda1add5b360837d0b3149"} Jul 07 14:38:38.551283 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.551272 2583 scope.go:117] "RemoveContainer" containerID="f2fbc69e372f51b083ac547bc1e344cd31a27b1db31fc3fc7883700830a36681" Jul 07 14:38:38.551488 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.551468 2583 scope.go:117] "RemoveContainer" containerID="81c411940b0188bdb3c007ab402e4a36f72b62ecf7cda1add5b360837d0b3149" Jul 07 14:38:38.551632 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:38.551615 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-x5rws_openshift-console-operator(6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30)\"" pod="openshift-console-operator/console-operator-68b757fb67-x5rws" podUID="6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30" Jul 07 14:38:38.574773 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.574750 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7de534e8-1ede-4999-9322-2944033cf5be-crio-socket\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:38.574887 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.574794 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7de534e8-1ede-4999-9322-2944033cf5be-data-volume\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:38.574887 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.574856 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:38.574887 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.574867 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7de534e8-1ede-4999-9322-2944033cf5be-crio-socket\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:38.575047 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.574899 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7de534e8-1ede-4999-9322-2944033cf5be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:38.575047 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.574950 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7lfk\" (UniqueName: \"kubernetes.io/projected/7de534e8-1ede-4999-9322-2944033cf5be-kube-api-access-v7lfk\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:38.575047 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:38.574973 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:38:38.575047 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:38.575030 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls podName:7de534e8-1ede-4999-9322-2944033cf5be nodeName:}" failed. No retries permitted until 2026-07-07 14:38:39.075012872 +0000 UTC m=+251.986206529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6944g" (UID: "7de534e8-1ede-4999-9322-2944033cf5be") : secret "insights-runtime-extractor-tls" not found Jul 07 14:38:38.575235 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.575094 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7de534e8-1ede-4999-9322-2944033cf5be-data-volume\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:38.575235 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:38.575166 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:38:38.575235 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:38.575227 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7de534e8-1ede-4999-9322-2944033cf5be-kube-rbac-proxy-cm podName:7de534e8-1ede-4999-9322-2944033cf5be nodeName:}" failed. No retries permitted until 2026-07-07 14:38:39.075213462 +0000 UTC m=+251.986407121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7de534e8-1ede-4999-9322-2944033cf5be-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6944g" (UID: "7de534e8-1ede-4999-9322-2944033cf5be") : configmap "kube-rbac-proxy" not found Jul 07 14:38:38.582825 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.582803 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7lfk\" (UniqueName: \"kubernetes.io/projected/7de534e8-1ede-4999-9322-2944033cf5be-kube-api-access-v7lfk\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:38.697725 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.697674 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:38.818674 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.818651 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:38.922621 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:38.922597 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:39.078353 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:39.078328 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:39.078456 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:39.078412 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7de534e8-1ede-4999-9322-2944033cf5be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:39.078456 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:39.078427 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:38:39.078526 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:39.078466 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:38:39.078526 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:39.078483 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7de534e8-1ede-4999-9322-2944033cf5be-kube-rbac-proxy-cm podName:7de534e8-1ede-4999-9322-2944033cf5be nodeName:}" failed. No retries permitted until 2026-07-07 14:38:40.078469971 +0000 UTC m=+252.989663628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7de534e8-1ede-4999-9322-2944033cf5be-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6944g" (UID: "7de534e8-1ede-4999-9322-2944033cf5be") : configmap "kube-rbac-proxy" not found Jul 07 14:38:39.078526 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:39.078512 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls podName:7de534e8-1ede-4999-9322-2944033cf5be nodeName:}" failed. No retries permitted until 2026-07-07 14:38:40.078499524 +0000 UTC m=+252.989693179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6944g" (UID: "7de534e8-1ede-4999-9322-2944033cf5be") : secret "insights-runtime-extractor-tls" not found Jul 07 14:38:39.298369 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:39.298322 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:39.381079 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:39.381022 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs\") pod \"network-metrics-daemon-xsfqs\" (UID: \"d60cabb9-29c9-4f9d-b147-2ca1844f5d96\") " pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:38:39.381167 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:39.381121 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 14:38:39.381167 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:39.381165 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs podName:d60cabb9-29c9-4f9d-b147-2ca1844f5d96 nodeName:}" failed. No retries permitted until 2026-07-07 14:40:41.381153555 +0000 UTC m=+374.292347212 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs") pod "network-metrics-daemon-xsfqs" (UID: "d60cabb9-29c9-4f9d-b147-2ca1844f5d96") : secret "metrics-daemon-secret" not found Jul 07 14:38:39.540571 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:39.540545 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:39.555269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:39.555247 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 14:38:39.555589 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:39.555554 2583 scope.go:117] "RemoveContainer" containerID="81c411940b0188bdb3c007ab402e4a36f72b62ecf7cda1add5b360837d0b3149" Jul 07 14:38:39.555732 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:39.555701 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-x5rws_openshift-console-operator(6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30)\"" pod="openshift-console-operator/console-operator-68b757fb67-x5rws" podUID="6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30" Jul 07 14:38:39.614450 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:39.614426 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:40.008412 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:40.008382 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:40.086211 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:40.086179 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:40.086323 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:40.086221 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7de534e8-1ede-4999-9322-2944033cf5be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:40.086323 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:40.086300 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:38:40.086459 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:40.086321 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:38:40.086459 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:40.086383 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7de534e8-1ede-4999-9322-2944033cf5be-kube-rbac-proxy-cm podName:7de534e8-1ede-4999-9322-2944033cf5be nodeName:}" failed. No retries permitted until 2026-07-07 14:38:42.086346038 +0000 UTC m=+254.997539705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7de534e8-1ede-4999-9322-2944033cf5be-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6944g" (UID: "7de534e8-1ede-4999-9322-2944033cf5be") : configmap "kube-rbac-proxy" not found Jul 07 14:38:40.086459 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:40.086403 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls podName:7de534e8-1ede-4999-9322-2944033cf5be nodeName:}" failed. No retries permitted until 2026-07-07 14:38:42.086393298 +0000 UTC m=+254.997586957 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6944g" (UID: "7de534e8-1ede-4999-9322-2944033cf5be") : secret "insights-runtime-extractor-tls" not found Jul 07 14:38:40.107672 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:40.107648 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:40.316764 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:40.316710 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rzkvv_1f877ac1-ec8e-437e-a186-28b499cf63e1/dns-node-resolver/0.log" Jul 07 14:38:41.116866 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.116832 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h78mx_80308b12-3ccc-477c-9578-1ac78d11db32/node-ca/0.log" Jul 07 14:38:41.531902 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.531865 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:41.599276 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.599247 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:41.603154 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.603130 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-cbhrv"] Jul 07 14:38:41.605798 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.605779 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-cbhrv" Jul 07 14:38:41.608871 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.608849 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 07 14:38:41.609475 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.609450 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rqjxs\"" Jul 07 14:38:41.609581 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.609530 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 07 14:38:41.609581 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.609530 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 07 14:38:41.609694 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.609534 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 07 14:38:41.614692 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.614664 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-cbhrv"] Jul 07 14:38:41.696783 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.696759 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5797a52-064d-4c81-ae66-454912e36619-signing-key\") pod \"service-ca-758f599c5-cbhrv\" (UID: \"b5797a52-064d-4c81-ae66-454912e36619\") " pod="openshift-service-ca/service-ca-758f599c5-cbhrv" Jul 07 14:38:41.696869 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.696786 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5797a52-064d-4c81-ae66-454912e36619-signing-cabundle\") pod \"service-ca-758f599c5-cbhrv\" (UID: \"b5797a52-064d-4c81-ae66-454912e36619\") " pod="openshift-service-ca/service-ca-758f599c5-cbhrv" Jul 07 14:38:41.696869 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.696807 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwjqd\" (UniqueName: \"kubernetes.io/projected/b5797a52-064d-4c81-ae66-454912e36619-kube-api-access-pwjqd\") pod \"service-ca-758f599c5-cbhrv\" (UID: \"b5797a52-064d-4c81-ae66-454912e36619\") " pod="openshift-service-ca/service-ca-758f599c5-cbhrv" Jul 07 14:38:41.718017 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.717990 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:41.797270 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.797217 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5797a52-064d-4c81-ae66-454912e36619-signing-key\") pod \"service-ca-758f599c5-cbhrv\" (UID: \"b5797a52-064d-4c81-ae66-454912e36619\") " pod="openshift-service-ca/service-ca-758f599c5-cbhrv" Jul 07 14:38:41.797270 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.797249 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5797a52-064d-4c81-ae66-454912e36619-signing-cabundle\") pod \"service-ca-758f599c5-cbhrv\" (UID: \"b5797a52-064d-4c81-ae66-454912e36619\") " pod="openshift-service-ca/service-ca-758f599c5-cbhrv" Jul 07 14:38:41.797270 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.797269 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwjqd\" (UniqueName: \"kubernetes.io/projected/b5797a52-064d-4c81-ae66-454912e36619-kube-api-access-pwjqd\") pod \"service-ca-758f599c5-cbhrv\" (UID: \"b5797a52-064d-4c81-ae66-454912e36619\") " pod="openshift-service-ca/service-ca-758f599c5-cbhrv" Jul 07 14:38:41.797847 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.797827 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5797a52-064d-4c81-ae66-454912e36619-signing-cabundle\") pod \"service-ca-758f599c5-cbhrv\" (UID: \"b5797a52-064d-4c81-ae66-454912e36619\") " pod="openshift-service-ca/service-ca-758f599c5-cbhrv" Jul 07 14:38:41.799485 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.799468 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5797a52-064d-4c81-ae66-454912e36619-signing-key\") pod \"service-ca-758f599c5-cbhrv\" (UID: \"b5797a52-064d-4c81-ae66-454912e36619\") " pod="openshift-service-ca/service-ca-758f599c5-cbhrv" Jul 07 14:38:41.806119 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.806095 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwjqd\" (UniqueName: \"kubernetes.io/projected/b5797a52-064d-4c81-ae66-454912e36619-kube-api-access-pwjqd\") pod \"service-ca-758f599c5-cbhrv\" (UID: \"b5797a52-064d-4c81-ae66-454912e36619\") " pod="openshift-service-ca/service-ca-758f599c5-cbhrv" Jul 07 14:38:41.819577 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.819554 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:41.914802 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.914783 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-cbhrv" Jul 07 14:38:41.915678 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:41.915665 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7dfs5_288a2a6a-6435-4eab-bb18-ff6cf90ec0be/serve-healthcheck-canary/0.log" Jul 07 14:38:42.034703 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:42.034676 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-cbhrv"] Jul 07 14:38:42.038165 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:38:42.038132 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5797a52_064d_4c81_ae66_454912e36619.slice/crio-1ad19d492706997713daebe379152b55c94a3bbf83f86eb9bf67bd962df68d77 WatchSource:0}: Error finding container 1ad19d492706997713daebe379152b55c94a3bbf83f86eb9bf67bd962df68d77: Status 404 returned error can't find the container with id 1ad19d492706997713daebe379152b55c94a3bbf83f86eb9bf67bd962df68d77 Jul 07 14:38:42.099166 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:42.099112 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:42.099166 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:42.099145 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7de534e8-1ede-4999-9322-2944033cf5be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:42.099284 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:42.099233 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:38:42.099284 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:42.099262 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:38:42.099284 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:42.099283 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7de534e8-1ede-4999-9322-2944033cf5be-kube-rbac-proxy-cm podName:7de534e8-1ede-4999-9322-2944033cf5be nodeName:}" failed. No retries permitted until 2026-07-07 14:38:46.09926853 +0000 UTC m=+259.010462186 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7de534e8-1ede-4999-9322-2944033cf5be-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6944g" (UID: "7de534e8-1ede-4999-9322-2944033cf5be") : configmap "kube-rbac-proxy" not found Jul 07 14:38:42.099402 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:42.099311 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls podName:7de534e8-1ede-4999-9322-2944033cf5be nodeName:}" failed. No retries permitted until 2026-07-07 14:38:46.099297043 +0000 UTC m=+259.010490703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6944g" (UID: "7de534e8-1ede-4999-9322-2944033cf5be") : secret "insights-runtime-extractor-tls" not found Jul 07 14:38:42.565669 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:42.565637 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-cbhrv" event={"ID":"b5797a52-064d-4c81-ae66-454912e36619","Type":"ContainerStarted","Data":"1ad19d492706997713daebe379152b55c94a3bbf83f86eb9bf67bd962df68d77"} Jul 07 14:38:42.657953 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:42.657929 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:44.572980 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:44.572944 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-cbhrv" event={"ID":"b5797a52-064d-4c81-ae66-454912e36619","Type":"ContainerStarted","Data":"1ceaeb87646b65646100cbe30f1b45ad73c0c9a5f646cde954f5d5705b8f75cc"} Jul 07 14:38:44.591390 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:44.591331 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-cbhrv" podStartSLOduration=1.2783108699999999 podStartE2EDuration="3.591315034s" totalImagesPullingTime="2.313004164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:38:44.590355543 +0000 UTC m=+257.501549220" watchObservedRunningTime="2026-07-07 14:38:44.591315034 +0000 UTC m=+257.502508713" Jul 07 14:38:44.788330 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:44.788293 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-x5rws" Jul 07 14:38:44.788526 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:44.788382 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-x5rws" Jul 07 14:38:44.788745 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:44.788729 2583 scope.go:117] "RemoveContainer" containerID="81c411940b0188bdb3c007ab402e4a36f72b62ecf7cda1add5b360837d0b3149" Jul 07 14:38:44.788913 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:44.788893 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-x5rws_openshift-console-operator(6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30)\"" pod="openshift-console-operator/console-operator-68b757fb67-x5rws" podUID="6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30" Jul 07 14:38:45.107223 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:45.107128 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:46.127499 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:46.127458 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:46.128090 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:46.127510 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7de534e8-1ede-4999-9322-2944033cf5be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:46.128090 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:46.127611 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:38:46.128090 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:46.127644 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:38:46.128090 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:46.127687 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7de534e8-1ede-4999-9322-2944033cf5be-kube-rbac-proxy-cm podName:7de534e8-1ede-4999-9322-2944033cf5be nodeName:}" failed. No retries permitted until 2026-07-07 14:38:54.127669388 +0000 UTC m=+267.038863057 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7de534e8-1ede-4999-9322-2944033cf5be-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6944g" (UID: "7de534e8-1ede-4999-9322-2944033cf5be") : configmap "kube-rbac-proxy" not found Jul 07 14:38:46.128090 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:46.127720 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls podName:7de534e8-1ede-4999-9322-2944033cf5be nodeName:}" failed. No retries permitted until 2026-07-07 14:38:54.127693629 +0000 UTC m=+267.038887296 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6944g" (UID: "7de534e8-1ede-4999-9322-2944033cf5be") : secret "insights-runtime-extractor-tls" not found Jul 07 14:38:46.189990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:46.189949 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:46.452426 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:46.452337 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:46.533906 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:46.533871 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:46.697196 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:46.697167 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:46.786935 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:46.786909 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:46.889901 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:46.889866 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:47.869996 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:47.869963 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:48.267101 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:48.267075 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:48.481936 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:48.481901 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:48.668750 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:48.668688 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:49.070605 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:49.070577 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:49.716852 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:49.716822 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:49.868589 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:49.868562 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:50.068601 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:50.068576 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:50.870755 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:50.870730 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:50.970461 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:50.970436 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:51.088816 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:51.088791 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:51.569082 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:51.569051 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:51.693992 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:51.693964 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:52.068447 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:52.068412 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:52.305629 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:52.305593 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:52.575926 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:52.575902 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:53.408814 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:53.408781 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:53.509326 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:53.509299 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:53.670051 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:53.669991 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:53.965036 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:53.964976 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:54.185613 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:54.185585 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:54.185724 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:54.185618 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7de534e8-1ede-4999-9322-2944033cf5be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:54.185763 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:54.185727 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:38:54.185813 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:38:54.185799 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls podName:7de534e8-1ede-4999-9322-2944033cf5be nodeName:}" failed. No retries permitted until 2026-07-07 14:39:10.185780316 +0000 UTC m=+283.096973984 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6944g" (UID: "7de534e8-1ede-4999-9322-2944033cf5be") : secret "insights-runtime-extractor-tls" not found Jul 07 14:38:54.186192 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:54.186174 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7de534e8-1ede-4999-9322-2944033cf5be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:38:54.266959 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:54.266936 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:54.461942 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:54.461918 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:54.670892 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:54.670841 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:55.071387 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:55.071324 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:55.878298 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:55.878265 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:56.144246 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:56.144175 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:56.281347 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:56.281316 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:56.668924 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:56.668892 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:56.865509 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:56.865457 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:58.786427 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:58.786395 2583 scope.go:117] "RemoveContainer" containerID="81c411940b0188bdb3c007ab402e4a36f72b62ecf7cda1add5b360837d0b3149" Jul 07 14:38:59.619713 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:59.619687 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 14:38:59.619946 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:59.619757 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-x5rws" event={"ID":"6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30","Type":"ContainerStarted","Data":"1fa88425776b4bdf5561505bab1b44c84a6fa2d20dbf1f053504c068be86606e"} Jul 07 14:38:59.620064 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:59.620042 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-x5rws" Jul 07 14:38:59.624396 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:59.624375 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-x5rws" Jul 07 14:38:59.643129 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:59.643086 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-x5rws" podStartSLOduration=23.884612993 podStartE2EDuration="25.643073764s" totalImagesPullingTime="1.758460771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:38:59.642480599 +0000 UTC m=+272.553674275" watchObservedRunningTime="2026-07-07 14:38:59.643073764 +0000 UTC m=+272.554267445" Jul 07 14:38:59.664312 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:59.664283 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:59.723471 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:59.723438 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:59.784516 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:59.784491 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:59.941076 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:38:59.941008 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:00.125663 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:00.125606 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:00.226278 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:00.226244 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:00.327996 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:00.327966 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:00.398875 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:00.398851 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:01.335670 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:01.335642 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:01.523452 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:01.523419 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:01.586952 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:01.586873 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:02.155303 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:02.155271 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:02.751773 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:02.751741 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:02.803958 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:02.803932 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:03.714099 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.714072 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d68c6fc7-8cmwn"] Jul 07 14:39:03.717079 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.717062 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.719745 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.719715 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 14:39:03.719856 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.719822 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tnl6d\"" Jul 07 14:39:03.719903 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.719853 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 14:39:03.720415 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.720397 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 14:39:03.725781 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.725762 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 14:39:03.730261 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.730242 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d68c6fc7-8cmwn"] Jul 07 14:39:03.750943 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.750923 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmvx4\" (UniqueName: \"kubernetes.io/projected/66c9385d-7cec-4bce-b51f-2affa3e0d21a-kube-api-access-bmvx4\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.751025 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.750953 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66c9385d-7cec-4bce-b51f-2affa3e0d21a-bound-sa-token\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.751025 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.750982 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66c9385d-7cec-4bce-b51f-2affa3e0d21a-trusted-ca\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.751101 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.751031 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/66c9385d-7cec-4bce-b51f-2affa3e0d21a-image-registry-private-configuration\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.751101 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.751068 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/66c9385d-7cec-4bce-b51f-2affa3e0d21a-registry-certificates\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.751164 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.751102 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/66c9385d-7cec-4bce-b51f-2affa3e0d21a-ca-trust-extracted\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.751164 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.751117 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/66c9385d-7cec-4bce-b51f-2affa3e0d21a-installation-pull-secrets\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.751227 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.751183 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66c9385d-7cec-4bce-b51f-2affa3e0d21a-registry-tls\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.809830 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.809809 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:03.851947 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.851924 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66c9385d-7cec-4bce-b51f-2affa3e0d21a-registry-tls\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.852028 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.851952 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmvx4\" (UniqueName: \"kubernetes.io/projected/66c9385d-7cec-4bce-b51f-2affa3e0d21a-kube-api-access-bmvx4\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.852028 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.851975 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66c9385d-7cec-4bce-b51f-2affa3e0d21a-bound-sa-token\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.852028 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.852004 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66c9385d-7cec-4bce-b51f-2affa3e0d21a-trusted-ca\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.852028 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.852024 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/66c9385d-7cec-4bce-b51f-2affa3e0d21a-image-registry-private-configuration\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.852235 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.852043 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/66c9385d-7cec-4bce-b51f-2affa3e0d21a-registry-certificates\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.852235 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.852085 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/66c9385d-7cec-4bce-b51f-2affa3e0d21a-ca-trust-extracted\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.852235 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.852101 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/66c9385d-7cec-4bce-b51f-2affa3e0d21a-installation-pull-secrets\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.852541 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.852518 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/66c9385d-7cec-4bce-b51f-2affa3e0d21a-ca-trust-extracted\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.853007 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.852985 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/66c9385d-7cec-4bce-b51f-2affa3e0d21a-trusted-ca\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.853209 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.853184 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/66c9385d-7cec-4bce-b51f-2affa3e0d21a-registry-certificates\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.854394 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.854355 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/66c9385d-7cec-4bce-b51f-2affa3e0d21a-installation-pull-secrets\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.854479 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.854428 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/66c9385d-7cec-4bce-b51f-2affa3e0d21a-registry-tls\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.854520 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.854499 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/66c9385d-7cec-4bce-b51f-2affa3e0d21a-image-registry-private-configuration\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.865197 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.865167 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmvx4\" (UniqueName: \"kubernetes.io/projected/66c9385d-7cec-4bce-b51f-2affa3e0d21a-kube-api-access-bmvx4\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.865821 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.865805 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/66c9385d-7cec-4bce-b51f-2affa3e0d21a-bound-sa-token\") pod \"image-registry-6d68c6fc7-8cmwn\" (UID: \"66c9385d-7cec-4bce-b51f-2affa3e0d21a\") " pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:03.924778 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:03.924754 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:04.025934 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:04.025913 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:04.147807 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:04.147778 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d68c6fc7-8cmwn"] Jul 07 14:39:04.150448 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:39:04.150424 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c9385d_7cec_4bce_b51f_2affa3e0d21a.slice/crio-92e4b205bd25a4684652ebd48101461a498083719087bd43d119691132c7aa57 WatchSource:0}: Error finding container 92e4b205bd25a4684652ebd48101461a498083719087bd43d119691132c7aa57: Status 404 returned error can't find the container with id 92e4b205bd25a4684652ebd48101461a498083719087bd43d119691132c7aa57 Jul 07 14:39:04.636007 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:04.635962 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" event={"ID":"66c9385d-7cec-4bce-b51f-2affa3e0d21a","Type":"ContainerStarted","Data":"428e9c84b7d9e084225d67af56c4e0c012b0866046e5523652818fe41e60a1bf"} Jul 07 14:39:04.636188 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:04.636021 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" event={"ID":"66c9385d-7cec-4bce-b51f-2affa3e0d21a","Type":"ContainerStarted","Data":"92e4b205bd25a4684652ebd48101461a498083719087bd43d119691132c7aa57"} Jul 07 14:39:04.636188 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:04.636097 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:04.659374 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:04.659318 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" podStartSLOduration=1.659305144 podStartE2EDuration="1.659305144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:39:04.657865485 +0000 UTC m=+277.569059162" watchObservedRunningTime="2026-07-07 14:39:04.659305144 +0000 UTC m=+277.570498822" Jul 07 14:39:05.355087 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:05.355044 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:05.972506 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:05.972472 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:06.040328 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:06.040301 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:06.419466 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:06.419386 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:06.475768 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:06.475741 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:07.133445 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:07.133401 2583 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" podUID="38d3d17e-c69d-4b8c-9c8f-4c1f318400ba" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 07 14:39:09.341240 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:09.341202 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:09.441601 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:09.441572 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:09.528504 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:09.528471 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:09.995345 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:09.995293 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:10.196298 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:10.196225 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:39:10.198335 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:10.198318 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7de534e8-1ede-4999-9322-2944033cf5be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6944g\" (UID: \"7de534e8-1ede-4999-9322-2944033cf5be\") " pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:39:10.296729 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:39:10.296695 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tdw9f" podUID="e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b" Jul 07 14:39:10.491455 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:10.491431 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6944g" Jul 07 14:39:10.619890 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:10.619865 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6944g"] Jul 07 14:39:10.622283 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:39:10.622253 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de534e8_1ede_4999_9322_2944033cf5be.slice/crio-adb0f4b36a94d563f141b09bf38f856d51a128f9f4efe188e246493b823cb9d6 WatchSource:0}: Error finding container adb0f4b36a94d563f141b09bf38f856d51a128f9f4efe188e246493b823cb9d6: Status 404 returned error can't find the container with id adb0f4b36a94d563f141b09bf38f856d51a128f9f4efe188e246493b823cb9d6 Jul 07 14:39:10.655942 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:10.655909 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6944g" event={"ID":"7de534e8-1ede-4999-9322-2944033cf5be","Type":"ContainerStarted","Data":"adb0f4b36a94d563f141b09bf38f856d51a128f9f4efe188e246493b823cb9d6"} Jul 07 14:39:10.655942 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:10.655935 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tdw9f" Jul 07 14:39:13.320202 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:13.320169 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:39:13.322595 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:13.322575 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b-metrics-tls\") pod \"dns-default-tdw9f\" (UID: \"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b\") " pod="openshift-dns/dns-default-tdw9f" Jul 07 14:39:13.358904 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:13.358878 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v5tsw\"" Jul 07 14:39:13.367074 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:13.367055 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tdw9f" Jul 07 14:39:13.552734 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:13.552683 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:13.631142 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:13.631064 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:13.883556 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:13.883461 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:14.428171 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:14.428150 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tdw9f"] Jul 07 14:39:14.430979 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:39:14.430954 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode239d9e6_6a5a_49c2_8b01_a80afbdf4f3b.slice/crio-de051eb05da834ede469b8787b95f910ac85f0967e74413deaca1c5adbd79875 WatchSource:0}: Error finding container de051eb05da834ede469b8787b95f910ac85f0967e74413deaca1c5adbd79875: Status 404 returned error can't find the container with id de051eb05da834ede469b8787b95f910ac85f0967e74413deaca1c5adbd79875 Jul 07 14:39:14.670501 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:14.670423 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6944g" event={"ID":"7de534e8-1ede-4999-9322-2944033cf5be","Type":"ContainerStarted","Data":"6b82814395bca889d4fe17ccaea2ca8e70fbe25f4e2193d54b965bdcc37e674e"} Jul 07 14:39:14.671302 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:14.671277 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tdw9f" event={"ID":"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b","Type":"ContainerStarted","Data":"de051eb05da834ede469b8787b95f910ac85f0967e74413deaca1c5adbd79875"} Jul 07 14:39:15.676789 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:15.676719 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6944g" event={"ID":"7de534e8-1ede-4999-9322-2944033cf5be","Type":"ContainerStarted","Data":"56e84906ee62dd6a2eecb17272d120fb970b7fbbd86379bcfa759f8aa7bc826b"} Jul 07 14:39:16.682514 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:16.682418 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tdw9f" event={"ID":"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b","Type":"ContainerStarted","Data":"05caf5504f966a4e028a9e2c2b883f91890c3b66444cd325a8077b35b9be6afb"} Jul 07 14:39:16.682514 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:16.682477 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tdw9f" event={"ID":"e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b","Type":"ContainerStarted","Data":"b0024297286e35da6b010cade5db82fe172a8ab8fd793390b9d05d6a9523bfa3"} Jul 07 14:39:16.682968 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:16.682567 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tdw9f" Jul 07 14:39:16.701251 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:16.701199 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tdw9f" podStartSLOduration=251.891524896 podStartE2EDuration="4m13.701184593s" totalImagesPullingTime="1.809659697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:39:16.700039806 +0000 UTC m=+289.611233484" watchObservedRunningTime="2026-07-07 14:39:16.701184593 +0000 UTC m=+289.612378271" Jul 07 14:39:16.793212 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:16.793183 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:16.849159 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:16.849083 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:17.133417 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:17.133377 2583 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" podUID="38d3d17e-c69d-4b8c-9c8f-4c1f318400ba" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 07 14:39:17.760477 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:17.760442 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:17.817599 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:17.817562 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:18.690979 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:18.690892 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6944g" event={"ID":"7de534e8-1ede-4999-9322-2944033cf5be","Type":"ContainerStarted","Data":"7ec9510512db3cdd005706090e4080fbb8d57f616732a0b5a248a580702dcebc"} Jul 07 14:39:18.709221 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:18.709174 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6944g" podStartSLOduration=33.151154491 podStartE2EDuration="40.709159158s" totalImagesPullingTime="7.558004667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:38:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:39:18.707956208 +0000 UTC m=+291.619149885" watchObservedRunningTime="2026-07-07 14:39:18.709159158 +0000 UTC m=+291.620352836" Jul 07 14:39:20.766029 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:20.765991 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:20.789687 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:39:20.789652 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xsfqs" podUID="d60cabb9-29c9-4f9d-b147-2ca1844f5d96" Jul 07 14:39:20.851021 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:20.850990 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:21.955952 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:21.955925 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:22.012290 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:22.012263 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:24.030194 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:24.030163 2583 patch_prober.go:28] interesting pod/image-registry-6d68c6fc7-8cmwn container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 14:39:24.030575 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:24.030209 2583 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" podUID="66c9385d-7cec-4bce-b51f-2affa3e0d21a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 14:39:24.720309 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:24.720278 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:24.806673 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:24.806645 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:25.548127 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:25.548098 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:25.643485 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:25.643460 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d68c6fc7-8cmwn" Jul 07 14:39:25.650420 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:25.650397 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:25.803903 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:25.803834 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:25.905914 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:25.905886 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:26.688640 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:26.688609 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tdw9f" Jul 07 14:39:26.809945 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:26.809916 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:26.872665 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:26.872628 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:27.134138 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.134104 2583 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" podUID="38d3d17e-c69d-4b8c-9c8f-4c1f318400ba" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 07 14:39:27.134290 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.134166 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" Jul 07 14:39:27.134689 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.134657 2583 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"1973d543a0ac5c43acd65773e80c16ebd9834af7ad23a133f1a24186dcf79b70"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 07 14:39:27.134746 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.134713 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" podUID="38d3d17e-c69d-4b8c-9c8f-4c1f318400ba" containerName="service-proxy" containerID="cri-o://1973d543a0ac5c43acd65773e80c16ebd9834af7ad23a133f1a24186dcf79b70" gracePeriod=30 Jul 07 14:39:27.517338 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.517317 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 14:39:27.517800 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.517775 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 14:39:27.533306 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.533191 2583 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 14:39:27.586604 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.586577 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:27.692907 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.692881 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:27.723042 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.723015 2583 generic.go:356] "Generic (PLEG): container finished" podID="38d3d17e-c69d-4b8c-9c8f-4c1f318400ba" containerID="1973d543a0ac5c43acd65773e80c16ebd9834af7ad23a133f1a24186dcf79b70" exitCode=2 Jul 07 14:39:27.723129 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.723065 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" event={"ID":"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba","Type":"ContainerDied","Data":"1973d543a0ac5c43acd65773e80c16ebd9834af7ad23a133f1a24186dcf79b70"} Jul 07 14:39:27.723129 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.723101 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6dd8bbdf4c-d9wqm" event={"ID":"38d3d17e-c69d-4b8c-9c8f-4c1f318400ba","Type":"ContainerStarted","Data":"e8b57f3867b78a9654bb2d2fd19e8a13a4a807988e91c710f7b06615a7dd744f"} Jul 07 14:39:27.785340 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.785271 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:27.911120 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.910957 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:27.940262 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.940237 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn"] Jul 07 14:39:27.943657 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.943636 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn" Jul 07 14:39:27.946127 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.946107 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 07 14:39:27.946235 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.946196 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 14:39:27.946399 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.946383 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 07 14:39:27.946481 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.946418 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 14:39:27.946688 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.946636 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-djz7n\"" Jul 07 14:39:27.946688 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.946660 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 14:39:27.957753 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.957733 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn"] Jul 07 14:39:27.978566 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.978542 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n7m8n"] Jul 07 14:39:27.981784 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.981767 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:27.984751 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.984733 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2kmmz\"" Jul 07 14:39:27.985128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.985109 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 14:39:27.985588 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.985570 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 14:39:27.985749 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:27.985721 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 14:39:28.014940 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.014920 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:28.021074 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.021049 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6a35f32-7606-4895-af52-5345da42bf07-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-kjndn\" (UID: \"b6a35f32-7606-4895-af52-5345da42bf07\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn" Jul 07 14:39:28.021156 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.021095 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw9t9\" (UniqueName: \"kubernetes.io/projected/b6a35f32-7606-4895-af52-5345da42bf07-kube-api-access-jw9t9\") pod \"openshift-state-metrics-54cf96dfcf-kjndn\" (UID: \"b6a35f32-7606-4895-af52-5345da42bf07\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn" Jul 07 14:39:28.021202 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.021152 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6a35f32-7606-4895-af52-5345da42bf07-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-kjndn\" (UID: \"b6a35f32-7606-4895-af52-5345da42bf07\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn" Jul 07 14:39:28.021273 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.021257 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6a35f32-7606-4895-af52-5345da42bf07-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-kjndn\" (UID: \"b6a35f32-7606-4895-af52-5345da42bf07\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn" Jul 07 14:39:28.088609 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.088557 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:28.122371 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.122334 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6a35f32-7606-4895-af52-5345da42bf07-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-kjndn\" (UID: \"b6a35f32-7606-4895-af52-5345da42bf07\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn" Jul 07 14:39:28.122454 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.122375 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw9t9\" (UniqueName: \"kubernetes.io/projected/b6a35f32-7606-4895-af52-5345da42bf07-kube-api-access-jw9t9\") pod \"openshift-state-metrics-54cf96dfcf-kjndn\" (UID: \"b6a35f32-7606-4895-af52-5345da42bf07\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn" Jul 07 14:39:28.122454 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.122394 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-node-exporter-tls\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.122454 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.122419 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6a35f32-7606-4895-af52-5345da42bf07-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-kjndn\" (UID: \"b6a35f32-7606-4895-af52-5345da42bf07\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn" Jul 07 14:39:28.122570 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.122460 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-sys\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.122570 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.122518 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgwc5\" (UniqueName: \"kubernetes.io/projected/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-kube-api-access-lgwc5\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.122570 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.122549 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.122716 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.122578 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-node-exporter-wtmp\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.122716 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.122622 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-metrics-client-ca\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.122716 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.122668 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-node-exporter-accelerators-collector-config\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.122716 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.122708 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6a35f32-7606-4895-af52-5345da42bf07-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-kjndn\" (UID: \"b6a35f32-7606-4895-af52-5345da42bf07\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn" Jul 07 14:39:28.122910 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.122732 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-root\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.122910 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.122766 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-node-exporter-textfile\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.123131 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.123111 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6a35f32-7606-4895-af52-5345da42bf07-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-kjndn\" (UID: \"b6a35f32-7606-4895-af52-5345da42bf07\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn" Jul 07 14:39:28.124808 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.124787 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6a35f32-7606-4895-af52-5345da42bf07-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-kjndn\" (UID: \"b6a35f32-7606-4895-af52-5345da42bf07\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn" Jul 07 14:39:28.124890 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.124814 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6a35f32-7606-4895-af52-5345da42bf07-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-kjndn\" (UID: \"b6a35f32-7606-4895-af52-5345da42bf07\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn" Jul 07 14:39:28.133066 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.133039 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw9t9\" (UniqueName: \"kubernetes.io/projected/b6a35f32-7606-4895-af52-5345da42bf07-kube-api-access-jw9t9\") pod \"openshift-state-metrics-54cf96dfcf-kjndn\" (UID: \"b6a35f32-7606-4895-af52-5345da42bf07\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn" Jul 07 14:39:28.185367 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.185331 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:28.223436 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.223411 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-node-exporter-tls\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.223532 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.223442 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-sys\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.223532 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.223484 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgwc5\" (UniqueName: \"kubernetes.io/projected/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-kube-api-access-lgwc5\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.223638 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.223539 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-sys\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.223638 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.223581 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.223638 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.223609 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-node-exporter-wtmp\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.223638 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.223634 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-metrics-client-ca\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.223849 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.223714 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-node-exporter-accelerators-collector-config\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.223849 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.223766 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-root\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.223849 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.223795 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-node-exporter-textfile\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.223849 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.223798 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-node-exporter-wtmp\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.224041 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.223861 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-root\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.224162 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.224137 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-node-exporter-textfile\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.224321 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.224299 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-node-exporter-accelerators-collector-config\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.224712 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.224694 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-metrics-client-ca\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.225674 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.225651 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-node-exporter-tls\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.225780 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.225700 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.232909 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.232885 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgwc5\" (UniqueName: \"kubernetes.io/projected/93c7f772-cc01-44d3-8ff2-f4c2bfedb630-kube-api-access-lgwc5\") pod \"node-exporter-n7m8n\" (UID: \"93c7f772-cc01-44d3-8ff2-f4c2bfedb630\") " pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.255700 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.255683 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn" Jul 07 14:39:28.285628 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.285595 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:28.291022 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.290999 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n7m8n" Jul 07 14:39:28.301473 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:39:28.301438 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c7f772_cc01_44d3_8ff2_f4c2bfedb630.slice/crio-fde12a3f2b1df680caa4e8462b0540f55f3dc809012f40d3aa40825397b6e55d WatchSource:0}: Error finding container fde12a3f2b1df680caa4e8462b0540f55f3dc809012f40d3aa40825397b6e55d: Status 404 returned error can't find the container with id fde12a3f2b1df680caa4e8462b0540f55f3dc809012f40d3aa40825397b6e55d Jul 07 14:39:28.394714 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.394685 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn"] Jul 07 14:39:28.401055 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:39:28.401028 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a35f32_7606_4895_af52_5345da42bf07.slice/crio-0c2459327f07f4039c8f8b0164f004fdac8a4c3e84577b45a29f90d72ef8ddb1 WatchSource:0}: Error finding container 0c2459327f07f4039c8f8b0164f004fdac8a4c3e84577b45a29f90d72ef8ddb1: Status 404 returned error can't find the container with id 0c2459327f07f4039c8f8b0164f004fdac8a4c3e84577b45a29f90d72ef8ddb1 Jul 07 14:39:28.413497 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.413476 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:28.516395 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.516375 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:28.727775 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.727732 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n7m8n" event={"ID":"93c7f772-cc01-44d3-8ff2-f4c2bfedb630","Type":"ContainerStarted","Data":"fde12a3f2b1df680caa4e8462b0540f55f3dc809012f40d3aa40825397b6e55d"} Jul 07 14:39:28.729468 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.729447 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn" event={"ID":"b6a35f32-7606-4895-af52-5345da42bf07","Type":"ContainerStarted","Data":"893aa256874f9a77ef8730f02524ae591faec40b16fcb295b802cd4d2b23adf8"} Jul 07 14:39:28.729579 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.729495 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn" event={"ID":"b6a35f32-7606-4895-af52-5345da42bf07","Type":"ContainerStarted","Data":"efebd9b3b3e8418419e8146b879ef7bbed7a3e08c7f418787858862d568e84b1"} Jul 07 14:39:28.729579 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.729517 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn" event={"ID":"b6a35f32-7606-4895-af52-5345da42bf07","Type":"ContainerStarted","Data":"0c2459327f07f4039c8f8b0164f004fdac8a4c3e84577b45a29f90d72ef8ddb1"} Jul 07 14:39:28.927082 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:28.927042 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:29.013422 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:29.013328 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:29.542072 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:29.542042 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:29.736158 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:29.736129 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn" event={"ID":"b6a35f32-7606-4895-af52-5345da42bf07","Type":"ContainerStarted","Data":"74071de1ed5660a7598d4a8c8acffd9d65474640169f2d5f9a04d5591a0a5a39"} Jul 07 14:39:29.737844 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:29.737812 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n7m8n" event={"ID":"93c7f772-cc01-44d3-8ff2-f4c2bfedb630","Type":"ContainerStarted","Data":"89d73ac5dc5cac9e43ce43b8077a8e023e432db07859481ad125226017ed37db"} Jul 07 14:39:29.755925 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:29.755887 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kjndn" podStartSLOduration=1.676576441 podStartE2EDuration="2.75587297s" totalImagesPullingTime="1.079296529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:39:29.755069539 +0000 UTC m=+302.666263227" watchObservedRunningTime="2026-07-07 14:39:29.75587297 +0000 UTC m=+302.667066647" Jul 07 14:39:30.224969 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:30.224938 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:30.324956 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:30.324924 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:30.425698 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:30.425668 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:30.742708 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:30.742670 2583 generic.go:356] "Generic (PLEG): container finished" podID="93c7f772-cc01-44d3-8ff2-f4c2bfedb630" containerID="89d73ac5dc5cac9e43ce43b8077a8e023e432db07859481ad125226017ed37db" exitCode=0 Jul 07 14:39:30.743077 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:30.742757 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n7m8n" event={"ID":"93c7f772-cc01-44d3-8ff2-f4c2bfedb630","Type":"ContainerDied","Data":"89d73ac5dc5cac9e43ce43b8077a8e023e432db07859481ad125226017ed37db"} Jul 07 14:39:30.935868 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:30.935831 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:31.014158 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:31.014085 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:31.112434 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:31.112406 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:31.195054 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:31.195030 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:31.748677 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:31.748644 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n7m8n" event={"ID":"93c7f772-cc01-44d3-8ff2-f4c2bfedb630","Type":"ContainerStarted","Data":"c0f09484b953d094fc4526f0de574ac1687e561c4120e810d45a949157e753ee"} Jul 07 14:39:31.749087 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:31.748738 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n7m8n" event={"ID":"93c7f772-cc01-44d3-8ff2-f4c2bfedb630","Type":"ContainerStarted","Data":"79890b539927c81978c3bed2fa3be1aa8d5c740733c33cc99012d8c7a8e7fdb5"} Jul 07 14:39:31.771048 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:31.771005 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n7m8n" podStartSLOduration=3.422084454 podStartE2EDuration="4.770991124s" totalImagesPullingTime="1.285502863s" totalInitContainerRuntime="63.403807ms" isStatefulPod=true podCreationTimestamp="2026-07-07 14:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:39:31.770144374 +0000 UTC m=+304.681338051" watchObservedRunningTime="2026-07-07 14:39:31.770991124 +0000 UTC m=+304.682184802" Jul 07 14:39:32.768396 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:32.768343 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:32.832285 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:32.832236 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:32.928338 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:32.928301 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:32.983691 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:32.983667 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:33.787055 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:33.787027 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:39:34.195550 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:34.195473 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:35.050990 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:35.050960 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:35.696039 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:35.695984 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:35.765552 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:35.765517 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:37.137598 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:37.137564 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:37.199498 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:37.199467 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:42.694613 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:42.694571 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:43.907175 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:43.907146 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:44.117635 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:44.117601 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:44.216824 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:44.216735 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:44.761292 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:44.761257 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:44.819823 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:44.819794 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:51.412370 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:51.412337 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:51.513045 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:39:51.513015 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:06.425521 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:06.425487 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:06.486419 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:06.486392 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:35.049923 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:35.049834 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:35.150302 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:35.150280 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:35.527786 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:35.527748 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:35.590352 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:35.590326 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:41.424815 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:41.424778 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs\") pod \"network-metrics-daemon-xsfqs\" (UID: \"d60cabb9-29c9-4f9d-b147-2ca1844f5d96\") " pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:40:41.427007 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:41.426983 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d60cabb9-29c9-4f9d-b147-2ca1844f5d96-metrics-certs\") pod \"network-metrics-daemon-xsfqs\" (UID: \"d60cabb9-29c9-4f9d-b147-2ca1844f5d96\") " pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:40:41.589655 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:41.589631 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v8hh9\"" Jul 07 14:40:41.597643 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:41.597624 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xsfqs" Jul 07 14:40:41.719813 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:41.719786 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xsfqs"] Jul 07 14:40:41.721948 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:40:41.721917 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60cabb9_29c9_4f9d_b147_2ca1844f5d96.slice/crio-4efa5a818b1e1e823d4ec5c82d769e62b13409d52af5c54b94bc31c1f19c975f WatchSource:0}: Error finding container 4efa5a818b1e1e823d4ec5c82d769e62b13409d52af5c54b94bc31c1f19c975f: Status 404 returned error can't find the container with id 4efa5a818b1e1e823d4ec5c82d769e62b13409d52af5c54b94bc31c1f19c975f Jul 07 14:40:41.723893 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:41.723878 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:40:41.968684 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:41.968651 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xsfqs" event={"ID":"d60cabb9-29c9-4f9d-b147-2ca1844f5d96","Type":"ContainerStarted","Data":"4efa5a818b1e1e823d4ec5c82d769e62b13409d52af5c54b94bc31c1f19c975f"} Jul 07 14:40:42.973813 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:42.973780 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xsfqs" event={"ID":"d60cabb9-29c9-4f9d-b147-2ca1844f5d96","Type":"ContainerStarted","Data":"be267257cce5777b7f8da7d386597b6d10b60a22896b3d041f3b9c975db4491d"} Jul 07 14:40:42.974222 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:42.973830 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xsfqs" event={"ID":"d60cabb9-29c9-4f9d-b147-2ca1844f5d96","Type":"ContainerStarted","Data":"d4879cb17501eb357bf6b1fd9690d647ab68dd0da6a4b8df9ca5f48198b3b03e"} Jul 07 14:40:42.989743 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:42.989698 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xsfqs" podStartSLOduration=373.948461798 podStartE2EDuration="6m14.989685606s" totalImagesPullingTime="1.041223808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:40:42.988943106 +0000 UTC m=+375.900136822" watchObservedRunningTime="2026-07-07 14:40:42.989685606 +0000 UTC m=+375.900879283" Jul 07 14:40:43.083483 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:43.083417 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:43.327253 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:43.327225 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:47.219459 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:47.219422 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:47.696050 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:47.695955 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:56.837156 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:56.837119 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:56.933732 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:56.933705 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:57.035302 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:57.035273 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:57.561774 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:57.561735 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:57.663067 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:40:57.663036 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:08.243951 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:41:08.243912 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:08.344464 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:41:08.344436 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:08.405839 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:41:08.405806 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:13.814295 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:41:13.814252 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:16.131989 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:41:16.131955 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:02.451486 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:42:02.451444 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:02.510941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:42:02.510908 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:14.323484 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:42:14.323447 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:14.377422 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:42:14.377394 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:21.433131 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:42:21.433089 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:21.534041 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:42:21.534014 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:27.950574 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:42:27.950541 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:28.035654 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:42:28.035624 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:28.110423 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:42:28.110394 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:28.698139 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:42:28.698107 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:38.606909 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:42:38.606866 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:38.708087 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:42:38.708056 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:38.769065 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:42:38.769041 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:00.439060 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:00.439021 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:00.538480 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:00.538453 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:01.213483 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:01.213449 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns"] Jul 07 14:43:01.216535 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:01.216511 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns" Jul 07 14:43:01.219043 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:01.219023 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 14:43:01.219879 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:01.219863 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-29pj4\"" Jul 07 14:43:01.219963 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:01.219870 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 14:43:01.226396 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:01.226372 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns"] Jul 07 14:43:01.344572 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:01.344530 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79bd13db-00ad-4726-a0d6-745822b1cc2f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns\" (UID: \"79bd13db-00ad-4726-a0d6-745822b1cc2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns" Jul 07 14:43:01.344572 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:01.344573 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79bd13db-00ad-4726-a0d6-745822b1cc2f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns\" (UID: \"79bd13db-00ad-4726-a0d6-745822b1cc2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns" Jul 07 14:43:01.344785 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:01.344625 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhkjk\" (UniqueName: \"kubernetes.io/projected/79bd13db-00ad-4726-a0d6-745822b1cc2f-kube-api-access-xhkjk\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns\" (UID: \"79bd13db-00ad-4726-a0d6-745822b1cc2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns" Jul 07 14:43:01.445225 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:01.445188 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhkjk\" (UniqueName: \"kubernetes.io/projected/79bd13db-00ad-4726-a0d6-745822b1cc2f-kube-api-access-xhkjk\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns\" (UID: \"79bd13db-00ad-4726-a0d6-745822b1cc2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns" Jul 07 14:43:01.445635 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:01.445266 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79bd13db-00ad-4726-a0d6-745822b1cc2f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns\" (UID: \"79bd13db-00ad-4726-a0d6-745822b1cc2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns" Jul 07 14:43:01.445635 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:01.445295 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79bd13db-00ad-4726-a0d6-745822b1cc2f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns\" (UID: \"79bd13db-00ad-4726-a0d6-745822b1cc2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns" Jul 07 14:43:01.445777 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:01.445754 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79bd13db-00ad-4726-a0d6-745822b1cc2f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns\" (UID: \"79bd13db-00ad-4726-a0d6-745822b1cc2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns" Jul 07 14:43:01.445817 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:01.445766 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79bd13db-00ad-4726-a0d6-745822b1cc2f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns\" (UID: \"79bd13db-00ad-4726-a0d6-745822b1cc2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns" Jul 07 14:43:01.453940 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:01.453915 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhkjk\" (UniqueName: \"kubernetes.io/projected/79bd13db-00ad-4726-a0d6-745822b1cc2f-kube-api-access-xhkjk\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns\" (UID: \"79bd13db-00ad-4726-a0d6-745822b1cc2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns" Jul 07 14:43:01.524634 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:01.524604 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns" Jul 07 14:43:01.650072 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:01.650041 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns"] Jul 07 14:43:01.653341 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:43:01.653312 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79bd13db_00ad_4726_a0d6_745822b1cc2f.slice/crio-bf9a42a2f0792c2764dfa71b8c119cc1977497e3fad441ab8e7117b16d79e2b0 WatchSource:0}: Error finding container bf9a42a2f0792c2764dfa71b8c119cc1977497e3fad441ab8e7117b16d79e2b0: Status 404 returned error can't find the container with id bf9a42a2f0792c2764dfa71b8c119cc1977497e3fad441ab8e7117b16d79e2b0 Jul 07 14:43:02.434073 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:02.434041 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns" event={"ID":"79bd13db-00ad-4726-a0d6-745822b1cc2f","Type":"ContainerStarted","Data":"bf9a42a2f0792c2764dfa71b8c119cc1977497e3fad441ab8e7117b16d79e2b0"} Jul 07 14:43:09.460697 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:09.460621 2583 generic.go:356] "Generic (PLEG): container finished" podID="79bd13db-00ad-4726-a0d6-745822b1cc2f" containerID="9ae00272e21aea655555a8352ea625bfc9c940b72594cf2086322b84b495870e" exitCode=0 Jul 07 14:43:09.460697 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:09.460691 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns" event={"ID":"79bd13db-00ad-4726-a0d6-745822b1cc2f","Type":"ContainerDied","Data":"9ae00272e21aea655555a8352ea625bfc9c940b72594cf2086322b84b495870e"} Jul 07 14:43:12.473184 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:12.473156 2583 generic.go:356] "Generic (PLEG): container finished" podID="79bd13db-00ad-4726-a0d6-745822b1cc2f" containerID="9ed7499c47f2301be3adebd397458cde70b59811f680ee3e9f2d263386d0ab06" exitCode=0 Jul 07 14:43:12.473553 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:12.473197 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns" event={"ID":"79bd13db-00ad-4726-a0d6-745822b1cc2f","Type":"ContainerDied","Data":"9ed7499c47f2301be3adebd397458cde70b59811f680ee3e9f2d263386d0ab06"} Jul 07 14:43:21.510490 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:21.510448 2583 generic.go:356] "Generic (PLEG): container finished" podID="79bd13db-00ad-4726-a0d6-745822b1cc2f" containerID="a9490d8105351c963d0b1c13a3298e826a3452aca5d0a6121ea886d8fcbdaea1" exitCode=0 Jul 07 14:43:21.510963 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:21.510495 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns" event={"ID":"79bd13db-00ad-4726-a0d6-745822b1cc2f","Type":"ContainerDied","Data":"a9490d8105351c963d0b1c13a3298e826a3452aca5d0a6121ea886d8fcbdaea1"} Jul 07 14:43:22.633266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:22.633243 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns" Jul 07 14:43:22.695274 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:22.695250 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhkjk\" (UniqueName: \"kubernetes.io/projected/79bd13db-00ad-4726-a0d6-745822b1cc2f-kube-api-access-xhkjk\") pod \"79bd13db-00ad-4726-a0d6-745822b1cc2f\" (UID: \"79bd13db-00ad-4726-a0d6-745822b1cc2f\") " Jul 07 14:43:22.695387 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:22.695313 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79bd13db-00ad-4726-a0d6-745822b1cc2f-util\") pod \"79bd13db-00ad-4726-a0d6-745822b1cc2f\" (UID: \"79bd13db-00ad-4726-a0d6-745822b1cc2f\") " Jul 07 14:43:22.695387 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:22.695337 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79bd13db-00ad-4726-a0d6-745822b1cc2f-bundle\") pod \"79bd13db-00ad-4726-a0d6-745822b1cc2f\" (UID: \"79bd13db-00ad-4726-a0d6-745822b1cc2f\") " Jul 07 14:43:22.696019 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:22.695996 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bd13db-00ad-4726-a0d6-745822b1cc2f-bundle" (OuterVolumeSpecName: "bundle") pod "79bd13db-00ad-4726-a0d6-745822b1cc2f" (UID: "79bd13db-00ad-4726-a0d6-745822b1cc2f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:43:22.697281 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:22.697262 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bd13db-00ad-4726-a0d6-745822b1cc2f-kube-api-access-xhkjk" (OuterVolumeSpecName: "kube-api-access-xhkjk") pod "79bd13db-00ad-4726-a0d6-745822b1cc2f" (UID: "79bd13db-00ad-4726-a0d6-745822b1cc2f"). InnerVolumeSpecName "kube-api-access-xhkjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:43:22.700478 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:22.700458 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bd13db-00ad-4726-a0d6-745822b1cc2f-util" (OuterVolumeSpecName: "util") pod "79bd13db-00ad-4726-a0d6-745822b1cc2f" (UID: "79bd13db-00ad-4726-a0d6-745822b1cc2f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:43:22.796302 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:22.796251 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79bd13db-00ad-4726-a0d6-745822b1cc2f-util\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:43:22.796302 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:22.796272 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79bd13db-00ad-4726-a0d6-745822b1cc2f-bundle\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:43:22.796302 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:22.796280 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhkjk\" (UniqueName: \"kubernetes.io/projected/79bd13db-00ad-4726-a0d6-745822b1cc2f-kube-api-access-xhkjk\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:43:23.519187 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:23.519150 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns" event={"ID":"79bd13db-00ad-4726-a0d6-745822b1cc2f","Type":"ContainerDied","Data":"bf9a42a2f0792c2764dfa71b8c119cc1977497e3fad441ab8e7117b16d79e2b0"} Jul 07 14:43:23.519396 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:23.519193 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tv5ns" Jul 07 14:43:23.519396 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:23.519197 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf9a42a2f0792c2764dfa71b8c119cc1977497e3fad441ab8e7117b16d79e2b0" Jul 07 14:43:24.599665 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:24.599635 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:27.283128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:27.283087 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:28.284508 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:28.284469 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:30.919303 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:30.919257 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:31.019466 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:31.019433 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:33.059033 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.059001 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:33.185438 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.185411 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:33.231281 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.231258 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k"] Jul 07 14:43:33.231520 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.231507 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79bd13db-00ad-4726-a0d6-745822b1cc2f" containerName="util" Jul 07 14:43:33.231561 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.231521 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="79bd13db-00ad-4726-a0d6-745822b1cc2f" containerName="util" Jul 07 14:43:33.231561 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.231530 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79bd13db-00ad-4726-a0d6-745822b1cc2f" containerName="pull" Jul 07 14:43:33.231561 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.231536 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="79bd13db-00ad-4726-a0d6-745822b1cc2f" containerName="pull" Jul 07 14:43:33.231561 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.231542 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79bd13db-00ad-4726-a0d6-745822b1cc2f" containerName="extract" Jul 07 14:43:33.231561 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.231547 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="79bd13db-00ad-4726-a0d6-745822b1cc2f" containerName="extract" Jul 07 14:43:33.231731 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.231588 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="79bd13db-00ad-4726-a0d6-745822b1cc2f" containerName="extract" Jul 07 14:43:33.235124 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.235106 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k" Jul 07 14:43:33.238458 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.238441 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-29pj4\"" Jul 07 14:43:33.238684 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.238671 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 14:43:33.238877 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.238864 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 14:43:33.249697 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.249676 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k"] Jul 07 14:43:33.300237 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.300212 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:33.368064 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.368013 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f044b34-59fd-45bc-99c3-ba330471680a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k\" (UID: \"3f044b34-59fd-45bc-99c3-ba330471680a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k" Jul 07 14:43:33.368064 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.368058 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8txgv\" (UniqueName: \"kubernetes.io/projected/3f044b34-59fd-45bc-99c3-ba330471680a-kube-api-access-8txgv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k\" (UID: \"3f044b34-59fd-45bc-99c3-ba330471680a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k" Jul 07 14:43:33.368186 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.368087 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f044b34-59fd-45bc-99c3-ba330471680a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k\" (UID: \"3f044b34-59fd-45bc-99c3-ba330471680a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k" Jul 07 14:43:33.469088 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.469064 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f044b34-59fd-45bc-99c3-ba330471680a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k\" (UID: \"3f044b34-59fd-45bc-99c3-ba330471680a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k" Jul 07 14:43:33.469195 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.469109 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8txgv\" (UniqueName: \"kubernetes.io/projected/3f044b34-59fd-45bc-99c3-ba330471680a-kube-api-access-8txgv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k\" (UID: \"3f044b34-59fd-45bc-99c3-ba330471680a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k" Jul 07 14:43:33.469257 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.469226 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f044b34-59fd-45bc-99c3-ba330471680a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k\" (UID: \"3f044b34-59fd-45bc-99c3-ba330471680a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k" Jul 07 14:43:33.469503 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.469484 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f044b34-59fd-45bc-99c3-ba330471680a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k\" (UID: \"3f044b34-59fd-45bc-99c3-ba330471680a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k" Jul 07 14:43:33.469559 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.469510 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f044b34-59fd-45bc-99c3-ba330471680a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k\" (UID: \"3f044b34-59fd-45bc-99c3-ba330471680a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k" Jul 07 14:43:33.483660 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.483639 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8txgv\" (UniqueName: \"kubernetes.io/projected/3f044b34-59fd-45bc-99c3-ba330471680a-kube-api-access-8txgv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k\" (UID: \"3f044b34-59fd-45bc-99c3-ba330471680a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k" Jul 07 14:43:33.544095 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.544064 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k" Jul 07 14:43:33.672305 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:33.672274 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k"] Jul 07 14:43:33.674827 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:43:33.674795 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f044b34_59fd_45bc_99c3_ba330471680a.slice/crio-4cc222f0ca335746c2ec957b73b476a8b41e93c773169164e20c815b294f4216 WatchSource:0}: Error finding container 4cc222f0ca335746c2ec957b73b476a8b41e93c773169164e20c815b294f4216: Status 404 returned error can't find the container with id 4cc222f0ca335746c2ec957b73b476a8b41e93c773169164e20c815b294f4216 Jul 07 14:43:34.562242 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:34.562212 2583 generic.go:356] "Generic (PLEG): container finished" podID="3f044b34-59fd-45bc-99c3-ba330471680a" containerID="1b02913b41cddcad3f0de1eace82ba50e22cf7c3a5aab7fecd315f1dd471c2c6" exitCode=0 Jul 07 14:43:34.562577 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:34.562301 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k" event={"ID":"3f044b34-59fd-45bc-99c3-ba330471680a","Type":"ContainerDied","Data":"1b02913b41cddcad3f0de1eace82ba50e22cf7c3a5aab7fecd315f1dd471c2c6"} Jul 07 14:43:34.562577 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:34.562355 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k" event={"ID":"3f044b34-59fd-45bc-99c3-ba330471680a","Type":"ContainerStarted","Data":"4cc222f0ca335746c2ec957b73b476a8b41e93c773169164e20c815b294f4216"} Jul 07 14:43:35.630566 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:35.630524 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:35.808587 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:35.808544 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:36.573321 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:36.573283 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k" event={"ID":"3f044b34-59fd-45bc-99c3-ba330471680a","Type":"ContainerStarted","Data":"1b9a1025276d7d3a961d87dbe49c67213ce0e562a59a57d24df1c129eea6afaa"} Jul 07 14:43:37.578573 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:37.578542 2583 generic.go:356] "Generic (PLEG): container finished" podID="3f044b34-59fd-45bc-99c3-ba330471680a" containerID="1b9a1025276d7d3a961d87dbe49c67213ce0e562a59a57d24df1c129eea6afaa" exitCode=0 Jul 07 14:43:37.578930 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:37.578589 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k" event={"ID":"3f044b34-59fd-45bc-99c3-ba330471680a","Type":"ContainerDied","Data":"1b9a1025276d7d3a961d87dbe49c67213ce0e562a59a57d24df1c129eea6afaa"} Jul 07 14:43:38.091111 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:38.091080 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:38.583963 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:38.583932 2583 generic.go:356] "Generic (PLEG): container finished" podID="3f044b34-59fd-45bc-99c3-ba330471680a" containerID="f4e80b5f4894154a03c97995f33ec6cde5a128550bcaf53d9cc9c295863c99fe" exitCode=0 Jul 07 14:43:38.584299 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:38.583996 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k" event={"ID":"3f044b34-59fd-45bc-99c3-ba330471680a","Type":"ContainerDied","Data":"f4e80b5f4894154a03c97995f33ec6cde5a128550bcaf53d9cc9c295863c99fe"} Jul 07 14:43:39.399865 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:39.399832 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:39.713322 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:39.713297 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k" Jul 07 14:43:39.813483 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:39.813461 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8txgv\" (UniqueName: \"kubernetes.io/projected/3f044b34-59fd-45bc-99c3-ba330471680a-kube-api-access-8txgv\") pod \"3f044b34-59fd-45bc-99c3-ba330471680a\" (UID: \"3f044b34-59fd-45bc-99c3-ba330471680a\") " Jul 07 14:43:39.813608 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:39.813543 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f044b34-59fd-45bc-99c3-ba330471680a-bundle\") pod \"3f044b34-59fd-45bc-99c3-ba330471680a\" (UID: \"3f044b34-59fd-45bc-99c3-ba330471680a\") " Jul 07 14:43:39.813608 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:39.813559 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f044b34-59fd-45bc-99c3-ba330471680a-util\") pod \"3f044b34-59fd-45bc-99c3-ba330471680a\" (UID: \"3f044b34-59fd-45bc-99c3-ba330471680a\") " Jul 07 14:43:39.813929 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:39.813904 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f044b34-59fd-45bc-99c3-ba330471680a-bundle" (OuterVolumeSpecName: "bundle") pod "3f044b34-59fd-45bc-99c3-ba330471680a" (UID: "3f044b34-59fd-45bc-99c3-ba330471680a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:43:39.815481 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:39.815452 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f044b34-59fd-45bc-99c3-ba330471680a-kube-api-access-8txgv" (OuterVolumeSpecName: "kube-api-access-8txgv") pod "3f044b34-59fd-45bc-99c3-ba330471680a" (UID: "3f044b34-59fd-45bc-99c3-ba330471680a"). InnerVolumeSpecName "kube-api-access-8txgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:43:39.818703 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:39.818683 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f044b34-59fd-45bc-99c3-ba330471680a-util" (OuterVolumeSpecName: "util") pod "3f044b34-59fd-45bc-99c3-ba330471680a" (UID: "3f044b34-59fd-45bc-99c3-ba330471680a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:43:39.914175 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:39.914144 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f044b34-59fd-45bc-99c3-ba330471680a-bundle\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:43:39.914175 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:39.914172 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f044b34-59fd-45bc-99c3-ba330471680a-util\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:43:39.914175 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:39.914181 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8txgv\" (UniqueName: \"kubernetes.io/projected/3f044b34-59fd-45bc-99c3-ba330471680a-kube-api-access-8txgv\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:43:40.592262 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:40.592228 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k" event={"ID":"3f044b34-59fd-45bc-99c3-ba330471680a","Type":"ContainerDied","Data":"4cc222f0ca335746c2ec957b73b476a8b41e93c773169164e20c815b294f4216"} Jul 07 14:43:40.592447 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:40.592275 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhtv5k" Jul 07 14:43:40.592447 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:40.592280 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cc222f0ca335746c2ec957b73b476a8b41e93c773169164e20c815b294f4216" Jul 07 14:43:43.196747 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.196712 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:43.345277 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.345246 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lswph"] Jul 07 14:43:43.345529 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.345516 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f044b34-59fd-45bc-99c3-ba330471680a" containerName="util" Jul 07 14:43:43.345573 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.345531 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f044b34-59fd-45bc-99c3-ba330471680a" containerName="util" Jul 07 14:43:43.345573 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.345540 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f044b34-59fd-45bc-99c3-ba330471680a" containerName="pull" Jul 07 14:43:43.345573 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.345546 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f044b34-59fd-45bc-99c3-ba330471680a" containerName="pull" Jul 07 14:43:43.345573 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.345556 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f044b34-59fd-45bc-99c3-ba330471680a" containerName="extract" Jul 07 14:43:43.345573 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.345562 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f044b34-59fd-45bc-99c3-ba330471680a" containerName="extract" Jul 07 14:43:43.345719 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.345603 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f044b34-59fd-45bc-99c3-ba330471680a" containerName="extract" Jul 07 14:43:43.348442 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.348426 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lswph" Jul 07 14:43:43.350880 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.350844 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 14:43:43.350880 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.350853 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 14:43:43.351524 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.351507 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-zmn4n\"" Jul 07 14:43:43.358565 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.358544 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lswph"] Jul 07 14:43:43.438551 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.438528 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64xfn\" (UniqueName: \"kubernetes.io/projected/863f4a48-3b8e-497a-865b-9d5dd7c7d4d9-kube-api-access-64xfn\") pod \"cert-manager-55b4d954d5-lswph\" (UID: \"863f4a48-3b8e-497a-865b-9d5dd7c7d4d9\") " pod="cert-manager/cert-manager-55b4d954d5-lswph" Jul 07 14:43:43.438662 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.438572 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/863f4a48-3b8e-497a-865b-9d5dd7c7d4d9-bound-sa-token\") pod \"cert-manager-55b4d954d5-lswph\" (UID: \"863f4a48-3b8e-497a-865b-9d5dd7c7d4d9\") " pod="cert-manager/cert-manager-55b4d954d5-lswph" Jul 07 14:43:43.538881 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.538845 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64xfn\" (UniqueName: \"kubernetes.io/projected/863f4a48-3b8e-497a-865b-9d5dd7c7d4d9-kube-api-access-64xfn\") pod \"cert-manager-55b4d954d5-lswph\" (UID: \"863f4a48-3b8e-497a-865b-9d5dd7c7d4d9\") " pod="cert-manager/cert-manager-55b4d954d5-lswph" Jul 07 14:43:43.538989 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.538897 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/863f4a48-3b8e-497a-865b-9d5dd7c7d4d9-bound-sa-token\") pod \"cert-manager-55b4d954d5-lswph\" (UID: \"863f4a48-3b8e-497a-865b-9d5dd7c7d4d9\") " pod="cert-manager/cert-manager-55b4d954d5-lswph" Jul 07 14:43:43.547503 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.547478 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/863f4a48-3b8e-497a-865b-9d5dd7c7d4d9-bound-sa-token\") pod \"cert-manager-55b4d954d5-lswph\" (UID: \"863f4a48-3b8e-497a-865b-9d5dd7c7d4d9\") " pod="cert-manager/cert-manager-55b4d954d5-lswph" Jul 07 14:43:43.547615 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.547597 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64xfn\" (UniqueName: \"kubernetes.io/projected/863f4a48-3b8e-497a-865b-9d5dd7c7d4d9-kube-api-access-64xfn\") pod \"cert-manager-55b4d954d5-lswph\" (UID: \"863f4a48-3b8e-497a-865b-9d5dd7c7d4d9\") " pod="cert-manager/cert-manager-55b4d954d5-lswph" Jul 07 14:43:43.656737 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.656718 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lswph" Jul 07 14:43:43.782162 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:43.782077 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lswph"] Jul 07 14:43:43.784792 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:43:43.784761 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod863f4a48_3b8e_497a_865b_9d5dd7c7d4d9.slice/crio-6b81fa6c0089e698ed13d3706202876a6cee9bcc51a610b03db767f939ece739 WatchSource:0}: Error finding container 6b81fa6c0089e698ed13d3706202876a6cee9bcc51a610b03db767f939ece739: Status 404 returned error can't find the container with id 6b81fa6c0089e698ed13d3706202876a6cee9bcc51a610b03db767f939ece739 Jul 07 14:43:44.607947 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:44.607900 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lswph" event={"ID":"863f4a48-3b8e-497a-865b-9d5dd7c7d4d9","Type":"ContainerStarted","Data":"6b81fa6c0089e698ed13d3706202876a6cee9bcc51a610b03db767f939ece739"} Jul 07 14:43:45.589287 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:45.589253 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:45.688855 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:45.688781 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:46.616898 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:46.616859 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lswph" event={"ID":"863f4a48-3b8e-497a-865b-9d5dd7c7d4d9","Type":"ContainerStarted","Data":"ed86f96bc2cb8b6aca8f02afbdd2e13a6b05ed4abf23219e768445d3d58847c0"} Jul 07 14:43:46.635214 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:46.635166 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-lswph" podStartSLOduration=1.205679722 podStartE2EDuration="3.635152915s" totalImagesPullingTime="2.429473193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:43:46.633451801 +0000 UTC m=+559.544645480" watchObservedRunningTime="2026-07-07 14:43:46.635152915 +0000 UTC m=+559.546346591" Jul 07 14:43:50.835967 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:50.835935 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:50.936255 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:50.936213 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:52.795770 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:52.795742 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj"] Jul 07 14:43:52.814605 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:52.814581 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj"] Jul 07 14:43:52.814740 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:52.814674 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj" Jul 07 14:43:52.817445 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:52.817418 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 14:43:52.818050 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:52.818027 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 14:43:52.818188 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:52.818098 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-29pj4\"" Jul 07 14:43:52.903979 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:52.903952 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc0a23b-8a82-4278-8110-2a7071fba5b4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj\" (UID: \"fbc0a23b-8a82-4278-8110-2a7071fba5b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj" Jul 07 14:43:52.904073 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:52.903992 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlf5h\" (UniqueName: \"kubernetes.io/projected/fbc0a23b-8a82-4278-8110-2a7071fba5b4-kube-api-access-mlf5h\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj\" (UID: \"fbc0a23b-8a82-4278-8110-2a7071fba5b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj" Jul 07 14:43:52.904073 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:52.904024 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc0a23b-8a82-4278-8110-2a7071fba5b4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj\" (UID: \"fbc0a23b-8a82-4278-8110-2a7071fba5b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj" Jul 07 14:43:52.962923 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:52.962894 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:53.004722 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:53.004696 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc0a23b-8a82-4278-8110-2a7071fba5b4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj\" (UID: \"fbc0a23b-8a82-4278-8110-2a7071fba5b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj" Jul 07 14:43:53.004803 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:53.004736 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlf5h\" (UniqueName: \"kubernetes.io/projected/fbc0a23b-8a82-4278-8110-2a7071fba5b4-kube-api-access-mlf5h\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj\" (UID: \"fbc0a23b-8a82-4278-8110-2a7071fba5b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj" Jul 07 14:43:53.004803 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:53.004769 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc0a23b-8a82-4278-8110-2a7071fba5b4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj\" (UID: \"fbc0a23b-8a82-4278-8110-2a7071fba5b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj" Jul 07 14:43:53.005057 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:53.005042 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc0a23b-8a82-4278-8110-2a7071fba5b4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj\" (UID: \"fbc0a23b-8a82-4278-8110-2a7071fba5b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj" Jul 07 14:43:53.005125 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:53.005107 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc0a23b-8a82-4278-8110-2a7071fba5b4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj\" (UID: \"fbc0a23b-8a82-4278-8110-2a7071fba5b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj" Jul 07 14:43:53.012508 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:53.012488 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlf5h\" (UniqueName: \"kubernetes.io/projected/fbc0a23b-8a82-4278-8110-2a7071fba5b4-kube-api-access-mlf5h\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj\" (UID: \"fbc0a23b-8a82-4278-8110-2a7071fba5b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj" Jul 07 14:43:53.124073 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:53.124024 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj" Jul 07 14:43:53.270876 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:53.266436 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj"] Jul 07 14:43:53.272321 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:43:53.272293 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc0a23b_8a82_4278_8110_2a7071fba5b4.slice/crio-0e8019a45c763dfffe5222029c0100a35bb66d053851a31e94fcbd7d91caa77d WatchSource:0}: Error finding container 0e8019a45c763dfffe5222029c0100a35bb66d053851a31e94fcbd7d91caa77d: Status 404 returned error can't find the container with id 0e8019a45c763dfffe5222029c0100a35bb66d053851a31e94fcbd7d91caa77d Jul 07 14:43:53.644073 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:53.644041 2583 generic.go:356] "Generic (PLEG): container finished" podID="fbc0a23b-8a82-4278-8110-2a7071fba5b4" containerID="e1aed1ab7d81bec948f53fc16a6ed3d0e24e6fdb45179f544bcf0b1c3049b41b" exitCode=0 Jul 07 14:43:53.644202 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:53.644084 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj" event={"ID":"fbc0a23b-8a82-4278-8110-2a7071fba5b4","Type":"ContainerDied","Data":"e1aed1ab7d81bec948f53fc16a6ed3d0e24e6fdb45179f544bcf0b1c3049b41b"} Jul 07 14:43:53.644202 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:53.644122 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj" event={"ID":"fbc0a23b-8a82-4278-8110-2a7071fba5b4","Type":"ContainerStarted","Data":"0e8019a45c763dfffe5222029c0100a35bb66d053851a31e94fcbd7d91caa77d"} Jul 07 14:43:55.653860 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:55.653816 2583 generic.go:356] "Generic (PLEG): container finished" podID="fbc0a23b-8a82-4278-8110-2a7071fba5b4" containerID="de83ff5a6dacc02fca7d5e2506cb97c8b38da7f62060b37917e7525b22821a41" exitCode=0 Jul 07 14:43:55.654248 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:55.653867 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj" event={"ID":"fbc0a23b-8a82-4278-8110-2a7071fba5b4","Type":"ContainerDied","Data":"de83ff5a6dacc02fca7d5e2506cb97c8b38da7f62060b37917e7525b22821a41"} Jul 07 14:43:56.660476 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:56.660441 2583 generic.go:356] "Generic (PLEG): container finished" podID="fbc0a23b-8a82-4278-8110-2a7071fba5b4" containerID="5f0a95344cfca772b2d7c9c42b0b70b11644c21230490fd2428a2dcdc258b2fa" exitCode=0 Jul 07 14:43:56.660844 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:56.660486 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj" event={"ID":"fbc0a23b-8a82-4278-8110-2a7071fba5b4","Type":"ContainerDied","Data":"5f0a95344cfca772b2d7c9c42b0b70b11644c21230490fd2428a2dcdc258b2fa"} Jul 07 14:43:57.790377 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:57.790338 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj" Jul 07 14:43:57.936874 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:57.936814 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlf5h\" (UniqueName: \"kubernetes.io/projected/fbc0a23b-8a82-4278-8110-2a7071fba5b4-kube-api-access-mlf5h\") pod \"fbc0a23b-8a82-4278-8110-2a7071fba5b4\" (UID: \"fbc0a23b-8a82-4278-8110-2a7071fba5b4\") " Jul 07 14:43:57.936874 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:57.936847 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc0a23b-8a82-4278-8110-2a7071fba5b4-util\") pod \"fbc0a23b-8a82-4278-8110-2a7071fba5b4\" (UID: \"fbc0a23b-8a82-4278-8110-2a7071fba5b4\") " Jul 07 14:43:57.937064 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:57.936893 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc0a23b-8a82-4278-8110-2a7071fba5b4-bundle\") pod \"fbc0a23b-8a82-4278-8110-2a7071fba5b4\" (UID: \"fbc0a23b-8a82-4278-8110-2a7071fba5b4\") " Jul 07 14:43:57.937616 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:57.937593 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc0a23b-8a82-4278-8110-2a7071fba5b4-bundle" (OuterVolumeSpecName: "bundle") pod "fbc0a23b-8a82-4278-8110-2a7071fba5b4" (UID: "fbc0a23b-8a82-4278-8110-2a7071fba5b4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:43:57.938842 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:57.938823 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc0a23b-8a82-4278-8110-2a7071fba5b4-kube-api-access-mlf5h" (OuterVolumeSpecName: "kube-api-access-mlf5h") pod "fbc0a23b-8a82-4278-8110-2a7071fba5b4" (UID: "fbc0a23b-8a82-4278-8110-2a7071fba5b4"). InnerVolumeSpecName "kube-api-access-mlf5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:43:57.942892 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:57.942869 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc0a23b-8a82-4278-8110-2a7071fba5b4-util" (OuterVolumeSpecName: "util") pod "fbc0a23b-8a82-4278-8110-2a7071fba5b4" (UID: "fbc0a23b-8a82-4278-8110-2a7071fba5b4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:43:58.038068 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:58.038033 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlf5h\" (UniqueName: \"kubernetes.io/projected/fbc0a23b-8a82-4278-8110-2a7071fba5b4-kube-api-access-mlf5h\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:43:58.038068 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:58.038066 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc0a23b-8a82-4278-8110-2a7071fba5b4-util\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:43:58.038068 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:58.038077 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc0a23b-8a82-4278-8110-2a7071fba5b4-bundle\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:43:58.670564 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:58.670525 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj" event={"ID":"fbc0a23b-8a82-4278-8110-2a7071fba5b4","Type":"ContainerDied","Data":"0e8019a45c763dfffe5222029c0100a35bb66d053851a31e94fcbd7d91caa77d"} Jul 07 14:43:58.670564 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:58.670549 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affzgfj" Jul 07 14:43:58.670770 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:58.670581 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e8019a45c763dfffe5222029c0100a35bb66d053851a31e94fcbd7d91caa77d" Jul 07 14:43:59.724251 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:43:59.724220 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:04.338663 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:04.338621 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:04.404056 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:04.404027 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:08.526270 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:08.526238 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:08.656754 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:08.656726 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:09.226443 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.226409 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:09.528305 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.528222 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:09.631618 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.631591 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:09.708131 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.708098 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz"] Jul 07 14:44:09.708399 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.708384 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbc0a23b-8a82-4278-8110-2a7071fba5b4" containerName="extract" Jul 07 14:44:09.708487 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.708402 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbc0a23b-8a82-4278-8110-2a7071fba5b4" containerName="extract" Jul 07 14:44:09.708487 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.708420 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbc0a23b-8a82-4278-8110-2a7071fba5b4" containerName="pull" Jul 07 14:44:09.708487 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.708432 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbc0a23b-8a82-4278-8110-2a7071fba5b4" containerName="pull" Jul 07 14:44:09.708487 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.708447 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbc0a23b-8a82-4278-8110-2a7071fba5b4" containerName="util" Jul 07 14:44:09.708487 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.708456 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbc0a23b-8a82-4278-8110-2a7071fba5b4" containerName="util" Jul 07 14:44:09.708714 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.708528 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbc0a23b-8a82-4278-8110-2a7071fba5b4" containerName="extract" Jul 07 14:44:09.710889 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.710870 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" Jul 07 14:44:09.716144 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.716118 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 14:44:09.716424 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.716404 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 14:44:09.716521 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.716415 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 14:44:09.716521 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.716427 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 14:44:09.716606 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.716557 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-g8lnk\"" Jul 07 14:44:09.761214 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.761196 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz"] Jul 07 14:44:09.804968 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.804914 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckx5x\" (UniqueName: \"kubernetes.io/projected/239c61dc-8cbd-4bca-8407-0f25210678b2-kube-api-access-ckx5x\") pod \"opendatahub-operator-controller-manager-655bb48ff6-rjktz\" (UID: \"239c61dc-8cbd-4bca-8407-0f25210678b2\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" Jul 07 14:44:09.804968 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.804951 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/239c61dc-8cbd-4bca-8407-0f25210678b2-manifests\") pod \"opendatahub-operator-controller-manager-655bb48ff6-rjktz\" (UID: \"239c61dc-8cbd-4bca-8407-0f25210678b2\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" Jul 07 14:44:09.805083 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.804977 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/239c61dc-8cbd-4bca-8407-0f25210678b2-webhook-cert\") pod \"opendatahub-operator-controller-manager-655bb48ff6-rjktz\" (UID: \"239c61dc-8cbd-4bca-8407-0f25210678b2\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" Jul 07 14:44:09.805083 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.805009 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/239c61dc-8cbd-4bca-8407-0f25210678b2-tmp\") pod \"opendatahub-operator-controller-manager-655bb48ff6-rjktz\" (UID: \"239c61dc-8cbd-4bca-8407-0f25210678b2\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" Jul 07 14:44:09.805083 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.805055 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/239c61dc-8cbd-4bca-8407-0f25210678b2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-655bb48ff6-rjktz\" (UID: \"239c61dc-8cbd-4bca-8407-0f25210678b2\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" Jul 07 14:44:09.906148 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.906122 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/239c61dc-8cbd-4bca-8407-0f25210678b2-webhook-cert\") pod \"opendatahub-operator-controller-manager-655bb48ff6-rjktz\" (UID: \"239c61dc-8cbd-4bca-8407-0f25210678b2\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" Jul 07 14:44:09.906220 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.906153 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/239c61dc-8cbd-4bca-8407-0f25210678b2-tmp\") pod \"opendatahub-operator-controller-manager-655bb48ff6-rjktz\" (UID: \"239c61dc-8cbd-4bca-8407-0f25210678b2\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" Jul 07 14:44:09.906220 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.906176 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/239c61dc-8cbd-4bca-8407-0f25210678b2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-655bb48ff6-rjktz\" (UID: \"239c61dc-8cbd-4bca-8407-0f25210678b2\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" Jul 07 14:44:09.906220 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.906205 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckx5x\" (UniqueName: \"kubernetes.io/projected/239c61dc-8cbd-4bca-8407-0f25210678b2-kube-api-access-ckx5x\") pod \"opendatahub-operator-controller-manager-655bb48ff6-rjktz\" (UID: \"239c61dc-8cbd-4bca-8407-0f25210678b2\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" Jul 07 14:44:09.906334 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.906242 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/239c61dc-8cbd-4bca-8407-0f25210678b2-manifests\") pod \"opendatahub-operator-controller-manager-655bb48ff6-rjktz\" (UID: \"239c61dc-8cbd-4bca-8407-0f25210678b2\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" Jul 07 14:44:09.906679 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.906660 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/239c61dc-8cbd-4bca-8407-0f25210678b2-manifests\") pod \"opendatahub-operator-controller-manager-655bb48ff6-rjktz\" (UID: \"239c61dc-8cbd-4bca-8407-0f25210678b2\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" Jul 07 14:44:09.908454 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.908434 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/239c61dc-8cbd-4bca-8407-0f25210678b2-tmp\") pod \"opendatahub-operator-controller-manager-655bb48ff6-rjktz\" (UID: \"239c61dc-8cbd-4bca-8407-0f25210678b2\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" Jul 07 14:44:09.908536 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.908501 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/239c61dc-8cbd-4bca-8407-0f25210678b2-webhook-cert\") pod \"opendatahub-operator-controller-manager-655bb48ff6-rjktz\" (UID: \"239c61dc-8cbd-4bca-8407-0f25210678b2\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" Jul 07 14:44:09.908661 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.908643 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/239c61dc-8cbd-4bca-8407-0f25210678b2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-655bb48ff6-rjktz\" (UID: \"239c61dc-8cbd-4bca-8407-0f25210678b2\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" Jul 07 14:44:09.918530 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.918511 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckx5x\" (UniqueName: \"kubernetes.io/projected/239c61dc-8cbd-4bca-8407-0f25210678b2-kube-api-access-ckx5x\") pod \"opendatahub-operator-controller-manager-655bb48ff6-rjktz\" (UID: \"239c61dc-8cbd-4bca-8407-0f25210678b2\") " pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" Jul 07 14:44:09.951779 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:09.951731 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:10.020485 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:10.020464 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" Jul 07 14:44:10.057063 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:10.056965 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:10.152737 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:10.152713 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz"] Jul 07 14:44:10.152930 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:10.152856 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:10.155163 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:44:10.155140 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239c61dc_8cbd_4bca_8407_0f25210678b2.slice/crio-70290b8a32518d0e5f77d5b2f160d8b674773742cb11c6744257bf3ada5b4cf5 WatchSource:0}: Error finding container 70290b8a32518d0e5f77d5b2f160d8b674773742cb11c6744257bf3ada5b4cf5: Status 404 returned error can't find the container with id 70290b8a32518d0e5f77d5b2f160d8b674773742cb11c6744257bf3ada5b4cf5 Jul 07 14:44:10.229682 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:10.229656 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:10.322681 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:10.322628 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:10.420626 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:10.420567 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:10.712749 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:10.712674 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" event={"ID":"239c61dc-8cbd-4bca-8407-0f25210678b2","Type":"ContainerStarted","Data":"70290b8a32518d0e5f77d5b2f160d8b674773742cb11c6744257bf3ada5b4cf5"} Jul 07 14:44:13.727666 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:13.727630 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" event={"ID":"239c61dc-8cbd-4bca-8407-0f25210678b2","Type":"ContainerStarted","Data":"09424eb8562b76a3a54c4a6bd065a38e3399518ed36820cc78c20fd6820e532c"} Jul 07 14:44:14.733755 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:14.733722 2583 generic.go:356] "Generic (PLEG): container finished" podID="239c61dc-8cbd-4bca-8407-0f25210678b2" containerID="09424eb8562b76a3a54c4a6bd065a38e3399518ed36820cc78c20fd6820e532c" exitCode=0 Jul 07 14:44:14.734186 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:14.733788 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" event={"ID":"239c61dc-8cbd-4bca-8407-0f25210678b2","Type":"ContainerDied","Data":"09424eb8562b76a3a54c4a6bd065a38e3399518ed36820cc78c20fd6820e532c"} Jul 07 14:44:15.739005 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:15.738973 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" event={"ID":"239c61dc-8cbd-4bca-8407-0f25210678b2","Type":"ContainerStarted","Data":"a3dd308e6ee3d99c7c1f9e82bf0130149461d2d3eabe7c2007030da8a134abda"} Jul 07 14:44:15.739383 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:15.739144 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" Jul 07 14:44:15.762536 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:15.762486 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" podStartSLOduration=1.927119232 podStartE2EDuration="6.762471937s" totalImagesPullingTime="3.797335655s" totalInitContainerRuntime="1.03801705s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:44:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:44:15.760836961 +0000 UTC m=+588.672030638" watchObservedRunningTime="2026-07-07 14:44:15.762471937 +0000 UTC m=+588.673665660" Jul 07 14:44:17.219646 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:17.219613 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:17.318515 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:17.318491 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:22.197126 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:22.197090 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:22.267265 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:22.267234 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:22.623740 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:22.623705 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:22.724079 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:22.724048 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:26.745764 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:26.745739 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-655bb48ff6-rjktz" Jul 07 14:44:27.545842 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:27.545807 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 14:44:27.546836 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:27.546811 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 14:44:28.897671 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:28.897639 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:29.025062 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:29.025031 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:29.823041 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:29.823008 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-87x9g"] Jul 07 14:44:29.825915 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:29.825899 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-87x9g" Jul 07 14:44:29.829392 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:29.829351 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 07 14:44:29.830186 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:29.830172 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-ldxjp\"" Jul 07 14:44:29.847073 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:29.847047 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7jg8\" (UniqueName: \"kubernetes.io/projected/5a37d477-1a1c-4084-8b6b-735c1e32a091-kube-api-access-r7jg8\") pod \"odh-model-controller-7998c67dc4-87x9g\" (UID: \"5a37d477-1a1c-4084-8b6b-735c1e32a091\") " pod="opendatahub/odh-model-controller-7998c67dc4-87x9g" Jul 07 14:44:29.847185 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:29.847091 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a37d477-1a1c-4084-8b6b-735c1e32a091-cert\") pod \"odh-model-controller-7998c67dc4-87x9g\" (UID: \"5a37d477-1a1c-4084-8b6b-735c1e32a091\") " pod="opendatahub/odh-model-controller-7998c67dc4-87x9g" Jul 07 14:44:29.848420 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:29.848401 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-87x9g"] Jul 07 14:44:29.916169 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:29.916130 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:29.947827 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:29.947788 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a37d477-1a1c-4084-8b6b-735c1e32a091-cert\") pod \"odh-model-controller-7998c67dc4-87x9g\" (UID: \"5a37d477-1a1c-4084-8b6b-735c1e32a091\") " pod="opendatahub/odh-model-controller-7998c67dc4-87x9g" Jul 07 14:44:29.947976 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:29.947868 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7jg8\" (UniqueName: \"kubernetes.io/projected/5a37d477-1a1c-4084-8b6b-735c1e32a091-kube-api-access-r7jg8\") pod \"odh-model-controller-7998c67dc4-87x9g\" (UID: \"5a37d477-1a1c-4084-8b6b-735c1e32a091\") " pod="opendatahub/odh-model-controller-7998c67dc4-87x9g" Jul 07 14:44:29.947976 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:44:29.947953 2583 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 07 14:44:29.948089 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:44:29.948029 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a37d477-1a1c-4084-8b6b-735c1e32a091-cert podName:5a37d477-1a1c-4084-8b6b-735c1e32a091 nodeName:}" failed. No retries permitted until 2026-07-07 14:44:30.448008983 +0000 UTC m=+603.359202646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5a37d477-1a1c-4084-8b6b-735c1e32a091-cert") pod "odh-model-controller-7998c67dc4-87x9g" (UID: "5a37d477-1a1c-4084-8b6b-735c1e32a091") : secret "odh-model-controller-webhook-cert" not found Jul 07 14:44:29.958209 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:29.958189 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7jg8\" (UniqueName: \"kubernetes.io/projected/5a37d477-1a1c-4084-8b6b-735c1e32a091-kube-api-access-r7jg8\") pod \"odh-model-controller-7998c67dc4-87x9g\" (UID: \"5a37d477-1a1c-4084-8b6b-735c1e32a091\") " pod="opendatahub/odh-model-controller-7998c67dc4-87x9g" Jul 07 14:44:30.009623 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.009586 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:30.134136 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.134050 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:30.173089 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.173067 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f6b8bcfd-vpbmb"] Jul 07 14:44:30.176070 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.176055 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f6b8bcfd-vpbmb" Jul 07 14:44:30.179138 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.179118 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-q8f7v\"" Jul 07 14:44:30.179247 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.179205 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 14:44:30.179483 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.179468 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 14:44:30.179694 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.179679 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 14:44:30.179746 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.179730 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 14:44:30.199108 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.199080 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f6b8bcfd-vpbmb"] Jul 07 14:44:30.250065 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.250041 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x79h5\" (UniqueName: \"kubernetes.io/projected/e28b42f3-a1c2-4fa1-9479-b20be56f35c3-kube-api-access-x79h5\") pod \"kube-auth-proxy-7f6b8bcfd-vpbmb\" (UID: \"e28b42f3-a1c2-4fa1-9479-b20be56f35c3\") " pod="openshift-ingress/kube-auth-proxy-7f6b8bcfd-vpbmb" Jul 07 14:44:30.250175 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.250131 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e28b42f3-a1c2-4fa1-9479-b20be56f35c3-tls-certs\") pod \"kube-auth-proxy-7f6b8bcfd-vpbmb\" (UID: \"e28b42f3-a1c2-4fa1-9479-b20be56f35c3\") " pod="openshift-ingress/kube-auth-proxy-7f6b8bcfd-vpbmb" Jul 07 14:44:30.250175 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.250165 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e28b42f3-a1c2-4fa1-9479-b20be56f35c3-tmp\") pod \"kube-auth-proxy-7f6b8bcfd-vpbmb\" (UID: \"e28b42f3-a1c2-4fa1-9479-b20be56f35c3\") " pod="openshift-ingress/kube-auth-proxy-7f6b8bcfd-vpbmb" Jul 07 14:44:30.261623 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.261594 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:30.351197 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.351172 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e28b42f3-a1c2-4fa1-9479-b20be56f35c3-tmp\") pod \"kube-auth-proxy-7f6b8bcfd-vpbmb\" (UID: \"e28b42f3-a1c2-4fa1-9479-b20be56f35c3\") " pod="openshift-ingress/kube-auth-proxy-7f6b8bcfd-vpbmb" Jul 07 14:44:30.351309 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.351204 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x79h5\" (UniqueName: \"kubernetes.io/projected/e28b42f3-a1c2-4fa1-9479-b20be56f35c3-kube-api-access-x79h5\") pod \"kube-auth-proxy-7f6b8bcfd-vpbmb\" (UID: \"e28b42f3-a1c2-4fa1-9479-b20be56f35c3\") " pod="openshift-ingress/kube-auth-proxy-7f6b8bcfd-vpbmb" Jul 07 14:44:30.351425 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.351403 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e28b42f3-a1c2-4fa1-9479-b20be56f35c3-tls-certs\") pod \"kube-auth-proxy-7f6b8bcfd-vpbmb\" (UID: \"e28b42f3-a1c2-4fa1-9479-b20be56f35c3\") " pod="openshift-ingress/kube-auth-proxy-7f6b8bcfd-vpbmb" Jul 07 14:44:30.353394 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.353354 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e28b42f3-a1c2-4fa1-9479-b20be56f35c3-tmp\") pod \"kube-auth-proxy-7f6b8bcfd-vpbmb\" (UID: \"e28b42f3-a1c2-4fa1-9479-b20be56f35c3\") " pod="openshift-ingress/kube-auth-proxy-7f6b8bcfd-vpbmb" Jul 07 14:44:30.353619 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.353602 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e28b42f3-a1c2-4fa1-9479-b20be56f35c3-tls-certs\") pod \"kube-auth-proxy-7f6b8bcfd-vpbmb\" (UID: \"e28b42f3-a1c2-4fa1-9479-b20be56f35c3\") " pod="openshift-ingress/kube-auth-proxy-7f6b8bcfd-vpbmb" Jul 07 14:44:30.360648 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.360626 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x79h5\" (UniqueName: \"kubernetes.io/projected/e28b42f3-a1c2-4fa1-9479-b20be56f35c3-kube-api-access-x79h5\") pod \"kube-auth-proxy-7f6b8bcfd-vpbmb\" (UID: \"e28b42f3-a1c2-4fa1-9479-b20be56f35c3\") " pod="openshift-ingress/kube-auth-proxy-7f6b8bcfd-vpbmb" Jul 07 14:44:30.452630 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.452556 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a37d477-1a1c-4084-8b6b-735c1e32a091-cert\") pod \"odh-model-controller-7998c67dc4-87x9g\" (UID: \"5a37d477-1a1c-4084-8b6b-735c1e32a091\") " pod="opendatahub/odh-model-controller-7998c67dc4-87x9g" Jul 07 14:44:30.454639 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.454622 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a37d477-1a1c-4084-8b6b-735c1e32a091-cert\") pod \"odh-model-controller-7998c67dc4-87x9g\" (UID: \"5a37d477-1a1c-4084-8b6b-735c1e32a091\") " pod="opendatahub/odh-model-controller-7998c67dc4-87x9g" Jul 07 14:44:30.485656 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.485628 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f6b8bcfd-vpbmb" Jul 07 14:44:30.623877 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.619638 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f6b8bcfd-vpbmb"] Jul 07 14:44:30.653485 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.653460 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9t5l7"] Jul 07 14:44:30.657300 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.657286 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-9t5l7" Jul 07 14:44:30.660438 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.660418 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 07 14:44:30.663213 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.663176 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:30.711546 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.711488 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9t5l7"] Jul 07 14:44:30.734782 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.734758 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-87x9g" Jul 07 14:44:30.753780 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.753754 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khzw7\" (UniqueName: \"kubernetes.io/projected/7a82903d-7b22-4a68-bc89-4d2b06b7d5d4-kube-api-access-khzw7\") pod \"model-serving-api-54d89bd5fc-9t5l7\" (UID: \"7a82903d-7b22-4a68-bc89-4d2b06b7d5d4\") " pod="opendatahub/model-serving-api-54d89bd5fc-9t5l7" Jul 07 14:44:30.753868 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.753792 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a82903d-7b22-4a68-bc89-4d2b06b7d5d4-tls-certs\") pod \"model-serving-api-54d89bd5fc-9t5l7\" (UID: \"7a82903d-7b22-4a68-bc89-4d2b06b7d5d4\") " pod="opendatahub/model-serving-api-54d89bd5fc-9t5l7" Jul 07 14:44:30.800580 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.800548 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f6b8bcfd-vpbmb" event={"ID":"e28b42f3-a1c2-4fa1-9479-b20be56f35c3","Type":"ContainerStarted","Data":"c911118ece2c0edc3b86ef110ccc264cb0c0d3cb2393d097416cb0a30e691137"} Jul 07 14:44:30.854295 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.854264 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khzw7\" (UniqueName: \"kubernetes.io/projected/7a82903d-7b22-4a68-bc89-4d2b06b7d5d4-kube-api-access-khzw7\") pod \"model-serving-api-54d89bd5fc-9t5l7\" (UID: \"7a82903d-7b22-4a68-bc89-4d2b06b7d5d4\") " pod="opendatahub/model-serving-api-54d89bd5fc-9t5l7" Jul 07 14:44:30.854508 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.854466 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a82903d-7b22-4a68-bc89-4d2b06b7d5d4-tls-certs\") pod \"model-serving-api-54d89bd5fc-9t5l7\" (UID: \"7a82903d-7b22-4a68-bc89-4d2b06b7d5d4\") " pod="opendatahub/model-serving-api-54d89bd5fc-9t5l7" Jul 07 14:44:30.854605 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:44:30.854585 2583 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 07 14:44:30.854674 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:44:30.854662 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a82903d-7b22-4a68-bc89-4d2b06b7d5d4-tls-certs podName:7a82903d-7b22-4a68-bc89-4d2b06b7d5d4 nodeName:}" failed. No retries permitted until 2026-07-07 14:44:31.354640372 +0000 UTC m=+604.265834035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7a82903d-7b22-4a68-bc89-4d2b06b7d5d4-tls-certs") pod "model-serving-api-54d89bd5fc-9t5l7" (UID: "7a82903d-7b22-4a68-bc89-4d2b06b7d5d4") : secret "model-serving-api-tls" not found Jul 07 14:44:30.871505 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.871485 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khzw7\" (UniqueName: \"kubernetes.io/projected/7a82903d-7b22-4a68-bc89-4d2b06b7d5d4-kube-api-access-khzw7\") pod \"model-serving-api-54d89bd5fc-9t5l7\" (UID: \"7a82903d-7b22-4a68-bc89-4d2b06b7d5d4\") " pod="opendatahub/model-serving-api-54d89bd5fc-9t5l7" Jul 07 14:44:30.883632 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.883608 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-87x9g"] Jul 07 14:44:30.885846 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:44:30.885823 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a37d477_1a1c_4084_8b6b_735c1e32a091.slice/crio-e1f1add4f8de4c039ae6bca9b160a1fd0f52464337a90faca73e689341893fc6 WatchSource:0}: Error finding container e1f1add4f8de4c039ae6bca9b160a1fd0f52464337a90faca73e689341893fc6: Status 404 returned error can't find the container with id e1f1add4f8de4c039ae6bca9b160a1fd0f52464337a90faca73e689341893fc6 Jul 07 14:44:30.957878 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:30.957843 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:31.356664 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:31.356624 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a82903d-7b22-4a68-bc89-4d2b06b7d5d4-tls-certs\") pod \"model-serving-api-54d89bd5fc-9t5l7\" (UID: \"7a82903d-7b22-4a68-bc89-4d2b06b7d5d4\") " pod="opendatahub/model-serving-api-54d89bd5fc-9t5l7" Jul 07 14:44:31.358971 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:31.358950 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a82903d-7b22-4a68-bc89-4d2b06b7d5d4-tls-certs\") pod \"model-serving-api-54d89bd5fc-9t5l7\" (UID: \"7a82903d-7b22-4a68-bc89-4d2b06b7d5d4\") " pod="opendatahub/model-serving-api-54d89bd5fc-9t5l7" Jul 07 14:44:31.566773 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:31.566743 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-9t5l7" Jul 07 14:44:31.725436 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:31.725401 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:31.760188 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:31.760144 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9t5l7"] Jul 07 14:44:31.807225 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:31.807191 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-9t5l7" event={"ID":"7a82903d-7b22-4a68-bc89-4d2b06b7d5d4","Type":"ContainerStarted","Data":"626cf3a84bcd0d4a49c9928fe0246d8290f15cb97fa400e335531324b04f75c0"} Jul 07 14:44:31.808596 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:31.808568 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-87x9g" event={"ID":"5a37d477-1a1c-4084-8b6b-735c1e32a091","Type":"ContainerStarted","Data":"e1f1add4f8de4c039ae6bca9b160a1fd0f52464337a90faca73e689341893fc6"} Jul 07 14:44:31.819227 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:31.819203 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:32.890825 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:32.890785 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:35.828286 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:35.828243 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f6b8bcfd-vpbmb" event={"ID":"e28b42f3-a1c2-4fa1-9479-b20be56f35c3","Type":"ContainerStarted","Data":"eab4beff7655f4b578b99cde6199bd31f6fe8b8ea02041afbb3f7ab44925ec4b"} Jul 07 14:44:35.829847 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:35.829822 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-87x9g" event={"ID":"5a37d477-1a1c-4084-8b6b-735c1e32a091","Type":"ContainerStarted","Data":"ddfee6722e2b420faf72fd0a5c977241a2a45086265e2fc80a1ac09b9a62b473"} Jul 07 14:44:35.829985 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:35.829943 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-87x9g" Jul 07 14:44:35.831184 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:35.831162 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-9t5l7" event={"ID":"7a82903d-7b22-4a68-bc89-4d2b06b7d5d4","Type":"ContainerStarted","Data":"b340c49283a97d2696e817ed46af24fb13474d27f7e884ff0c73877d0e5bf52c"} Jul 07 14:44:35.831302 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:35.831277 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-9t5l7" Jul 07 14:44:35.846076 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:35.846018 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f6b8bcfd-vpbmb" podStartSLOduration=1.5090103190000002 podStartE2EDuration="5.84600255s" totalImagesPullingTime="4.336992231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:44:35.845127098 +0000 UTC m=+608.756320787" watchObservedRunningTime="2026-07-07 14:44:35.84600255 +0000 UTC m=+608.757196229" Jul 07 14:44:35.864956 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:35.864900 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-87x9g" podStartSLOduration=2.794745346 podStartE2EDuration="6.864881367s" totalImagesPullingTime="4.070136021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:44:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:44:35.864218658 +0000 UTC m=+608.775412336" watchObservedRunningTime="2026-07-07 14:44:35.864881367 +0000 UTC m=+608.776075052" Jul 07 14:44:35.882719 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:35.882670 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-9t5l7" podStartSLOduration=2.687589708 podStartE2EDuration="5.88265384s" totalImagesPullingTime="3.195064132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:44:35.881626681 +0000 UTC m=+608.792820362" watchObservedRunningTime="2026-07-07 14:44:35.88265384 +0000 UTC m=+608.793847519" Jul 07 14:44:36.324873 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.324842 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:36.657955 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.657873 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2"] Jul 07 14:44:36.661160 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.661142 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.666861 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.666830 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-84rhl\"" Jul 07 14:44:36.677235 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.677215 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 14:44:36.689012 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.688987 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2"] Jul 07 14:44:36.697953 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.697919 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.698072 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.697962 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.698072 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.698003 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.698191 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.698071 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.698191 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.698105 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.698191 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.698129 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.698191 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.698160 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.698412 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.698205 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqh8c\" (UniqueName: \"kubernetes.io/projected/78f9ed39-b885-4069-8827-d7441f1ab19f-kube-api-access-tqh8c\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.698412 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.698245 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78f9ed39-b885-4069-8827-d7441f1ab19f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.701534 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.701511 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:36.799133 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.799100 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqh8c\" (UniqueName: \"kubernetes.io/projected/78f9ed39-b885-4069-8827-d7441f1ab19f-kube-api-access-tqh8c\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.799300 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.799140 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78f9ed39-b885-4069-8827-d7441f1ab19f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.799300 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.799168 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.799300 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.799187 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.799300 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.799202 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.799300 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.799226 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.799300 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.799253 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.799300 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.799270 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.799300 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.799288 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.799729 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.799632 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.799825 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.799793 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.799883 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.799863 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.799974 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.799955 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.800029 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.800014 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78f9ed39-b885-4069-8827-d7441f1ab19f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.801519 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.801500 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.801807 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.801787 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.809635 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.809609 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqh8c\" (UniqueName: \"kubernetes.io/projected/78f9ed39-b885-4069-8827-d7441f1ab19f-kube-api-access-tqh8c\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.811451 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.811431 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-548557588-qgbz2\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:36.845128 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.845106 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:36.950242 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.950150 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:36.975727 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:36.975692 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:37.111090 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:37.111068 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2"] Jul 07 14:44:37.112719 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:44:37.112696 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f9ed39_b885_4069_8827_d7441f1ab19f.slice/crio-6c45e81db42d44ae99f8797e3cb0e1a28af1f349f74285b47e70f7127deeb332 WatchSource:0}: Error finding container 6c45e81db42d44ae99f8797e3cb0e1a28af1f349f74285b47e70f7127deeb332: Status 404 returned error can't find the container with id 6c45e81db42d44ae99f8797e3cb0e1a28af1f349f74285b47e70f7127deeb332 Jul 07 14:44:37.842716 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:37.842682 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" event={"ID":"78f9ed39-b885-4069-8827-d7441f1ab19f","Type":"ContainerStarted","Data":"6c45e81db42d44ae99f8797e3cb0e1a28af1f349f74285b47e70f7127deeb332"} Jul 07 14:44:39.219281 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.219245 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls"] Jul 07 14:44:39.226187 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.226165 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls" Jul 07 14:44:39.230020 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.229995 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 14:44:39.231026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.231006 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 14:44:39.231026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.231025 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 14:44:39.231195 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.231114 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 14:44:39.231195 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.231156 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8hv5q\"" Jul 07 14:44:39.231292 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.231246 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:44:39.237286 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.237263 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls"] Jul 07 14:44:39.318099 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.318058 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1744a35d-94a1-40f5-91fc-0e6feb95cc29-metrics-cert\") pod \"lws-controller-manager-6d85f8474c-9wnls\" (UID: \"1744a35d-94a1-40f5-91fc-0e6feb95cc29\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls" Jul 07 14:44:39.318269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.318124 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1744a35d-94a1-40f5-91fc-0e6feb95cc29-manager-config\") pod \"lws-controller-manager-6d85f8474c-9wnls\" (UID: \"1744a35d-94a1-40f5-91fc-0e6feb95cc29\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls" Jul 07 14:44:39.318269 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.318227 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbfj7\" (UniqueName: \"kubernetes.io/projected/1744a35d-94a1-40f5-91fc-0e6feb95cc29-kube-api-access-lbfj7\") pod \"lws-controller-manager-6d85f8474c-9wnls\" (UID: \"1744a35d-94a1-40f5-91fc-0e6feb95cc29\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls" Jul 07 14:44:39.318427 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.318273 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1744a35d-94a1-40f5-91fc-0e6feb95cc29-cert\") pod \"lws-controller-manager-6d85f8474c-9wnls\" (UID: \"1744a35d-94a1-40f5-91fc-0e6feb95cc29\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls" Jul 07 14:44:39.419563 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.419519 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbfj7\" (UniqueName: \"kubernetes.io/projected/1744a35d-94a1-40f5-91fc-0e6feb95cc29-kube-api-access-lbfj7\") pod \"lws-controller-manager-6d85f8474c-9wnls\" (UID: \"1744a35d-94a1-40f5-91fc-0e6feb95cc29\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls" Jul 07 14:44:39.419748 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.419587 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1744a35d-94a1-40f5-91fc-0e6feb95cc29-cert\") pod \"lws-controller-manager-6d85f8474c-9wnls\" (UID: \"1744a35d-94a1-40f5-91fc-0e6feb95cc29\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls" Jul 07 14:44:39.419748 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.419631 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1744a35d-94a1-40f5-91fc-0e6feb95cc29-metrics-cert\") pod \"lws-controller-manager-6d85f8474c-9wnls\" (UID: \"1744a35d-94a1-40f5-91fc-0e6feb95cc29\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls" Jul 07 14:44:39.419748 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.419680 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1744a35d-94a1-40f5-91fc-0e6feb95cc29-manager-config\") pod \"lws-controller-manager-6d85f8474c-9wnls\" (UID: \"1744a35d-94a1-40f5-91fc-0e6feb95cc29\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls" Jul 07 14:44:39.420496 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.420471 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1744a35d-94a1-40f5-91fc-0e6feb95cc29-manager-config\") pod \"lws-controller-manager-6d85f8474c-9wnls\" (UID: \"1744a35d-94a1-40f5-91fc-0e6feb95cc29\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls" Jul 07 14:44:39.422440 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.422414 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1744a35d-94a1-40f5-91fc-0e6feb95cc29-cert\") pod \"lws-controller-manager-6d85f8474c-9wnls\" (UID: \"1744a35d-94a1-40f5-91fc-0e6feb95cc29\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls" Jul 07 14:44:39.422574 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.422552 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1744a35d-94a1-40f5-91fc-0e6feb95cc29-metrics-cert\") pod \"lws-controller-manager-6d85f8474c-9wnls\" (UID: \"1744a35d-94a1-40f5-91fc-0e6feb95cc29\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls" Jul 07 14:44:39.433791 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.433769 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbfj7\" (UniqueName: \"kubernetes.io/projected/1744a35d-94a1-40f5-91fc-0e6feb95cc29-kube-api-access-lbfj7\") pod \"lws-controller-manager-6d85f8474c-9wnls\" (UID: \"1744a35d-94a1-40f5-91fc-0e6feb95cc29\") " pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls" Jul 07 14:44:39.537054 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.536963 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls" Jul 07 14:44:39.806240 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.806200 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 14:44:39.806338 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.806272 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 14:44:39.806338 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.806300 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 14:44:39.870218 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:39.870196 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls"] Jul 07 14:44:39.872519 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:44:39.872494 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1744a35d_94a1_40f5_91fc_0e6feb95cc29.slice/crio-0a11928276053ab48f4dda0aad2bd5c1ebef1c48286bcb2a51b0c94dadd84eb7 WatchSource:0}: Error finding container 0a11928276053ab48f4dda0aad2bd5c1ebef1c48286bcb2a51b0c94dadd84eb7: Status 404 returned error can't find the container with id 0a11928276053ab48f4dda0aad2bd5c1ebef1c48286bcb2a51b0c94dadd84eb7 Jul 07 14:44:40.855958 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:40.855924 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" event={"ID":"78f9ed39-b885-4069-8827-d7441f1ab19f","Type":"ContainerStarted","Data":"294a1ed3fb35e84fcb0df670967729fbd8f6c3e9f5c4b2bf0d0b2c8d3e3ed7d7"} Jul 07 14:44:40.857048 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:40.857024 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls" event={"ID":"1744a35d-94a1-40f5-91fc-0e6feb95cc29","Type":"ContainerStarted","Data":"0a11928276053ab48f4dda0aad2bd5c1ebef1c48286bcb2a51b0c94dadd84eb7"} Jul 07 14:44:40.892026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:40.891712 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" podStartSLOduration=2.200478287 podStartE2EDuration="4.891695062s" totalImagesPullingTime="2.691216775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:44:40.889924015 +0000 UTC m=+613.801117694" watchObservedRunningTime="2026-07-07 14:44:40.891695062 +0000 UTC m=+613.802888757" Jul 07 14:44:40.976290 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:40.976223 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:40.977444 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:40.977415 2583 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-548557588-qgbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body= Jul 07 14:44:40.977556 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:40.977466 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" podUID="78f9ed39-b885-4069-8827-d7441f1ab19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" Jul 07 14:44:41.005368 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:41.005322 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:41.166538 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:41.166454 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:41.300382 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:41.300330 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:41.487543 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:41.487510 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2"] Jul 07 14:44:41.489497 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:41.489468 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:41.977031 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:41.976998 2583 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-548557588-qgbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body= Jul 07 14:44:41.977454 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:41.977050 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" podUID="78f9ed39-b885-4069-8827-d7441f1ab19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" Jul 07 14:44:42.314482 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:42.314405 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:42.424219 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:42.424187 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:42.560524 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:42.560494 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:42.866930 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:42.866898 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls" event={"ID":"1744a35d-94a1-40f5-91fc-0e6feb95cc29","Type":"ContainerStarted","Data":"dc64b70d9c283f695222f66e17d68990197ebb856a61dd40e6d2fd489ca39553"} Jul 07 14:44:42.867099 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:42.866990 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls" Jul 07 14:44:42.867234 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:42.867212 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" podUID="78f9ed39-b885-4069-8827-d7441f1ab19f" containerName="istio-proxy" containerID="cri-o://294a1ed3fb35e84fcb0df670967729fbd8f6c3e9f5c4b2bf0d0b2c8d3e3ed7d7" gracePeriod=30 Jul 07 14:44:42.890732 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:42.890688 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls" podStartSLOduration=1.629600919 podStartE2EDuration="3.890675151s" totalImagesPullingTime="2.261074232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:44:42.888044672 +0000 UTC m=+615.799238350" watchObservedRunningTime="2026-07-07 14:44:42.890675151 +0000 UTC m=+615.801868827" Jul 07 14:44:45.320502 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:45.320465 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:46.837968 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:46.837938 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-87x9g" Jul 07 14:44:46.839933 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:46.839915 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-9t5l7" Jul 07 14:44:47.344841 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:47.344806 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:48.125777 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.125712 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:48.197937 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.197900 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-envoy\") pod \"78f9ed39-b885-4069-8827-d7441f1ab19f\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " Jul 07 14:44:48.197937 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.197938 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-credential-socket\") pod \"78f9ed39-b885-4069-8827-d7441f1ab19f\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " Jul 07 14:44:48.198157 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.197966 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-workload-certs\") pod \"78f9ed39-b885-4069-8827-d7441f1ab19f\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " Jul 07 14:44:48.198157 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.197991 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqh8c\" (UniqueName: \"kubernetes.io/projected/78f9ed39-b885-4069-8827-d7441f1ab19f-kube-api-access-tqh8c\") pod \"78f9ed39-b885-4069-8827-d7441f1ab19f\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " Jul 07 14:44:48.198157 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.198018 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-token\") pod \"78f9ed39-b885-4069-8827-d7441f1ab19f\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " Jul 07 14:44:48.198157 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.198041 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-workload-socket\") pod \"78f9ed39-b885-4069-8827-d7441f1ab19f\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " Jul 07 14:44:48.198157 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.198065 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78f9ed39-b885-4069-8827-d7441f1ab19f-istiod-ca-cert\") pod \"78f9ed39-b885-4069-8827-d7441f1ab19f\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " Jul 07 14:44:48.198157 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.198138 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-podinfo\") pod \"78f9ed39-b885-4069-8827-d7441f1ab19f\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " Jul 07 14:44:48.198494 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.198165 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-data\") pod \"78f9ed39-b885-4069-8827-d7441f1ab19f\" (UID: \"78f9ed39-b885-4069-8827-d7441f1ab19f\") " Jul 07 14:44:48.198494 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.198210 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "78f9ed39-b885-4069-8827-d7441f1ab19f" (UID: "78f9ed39-b885-4069-8827-d7441f1ab19f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:44:48.198494 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.198253 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "78f9ed39-b885-4069-8827-d7441f1ab19f" (UID: "78f9ed39-b885-4069-8827-d7441f1ab19f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:44:48.198494 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.198336 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "78f9ed39-b885-4069-8827-d7441f1ab19f" (UID: "78f9ed39-b885-4069-8827-d7441f1ab19f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:44:48.198494 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.198471 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-credential-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:44:48.198713 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.198498 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-workload-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:44:48.198713 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.198513 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-workload-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:44:48.198713 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.198510 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78f9ed39-b885-4069-8827-d7441f1ab19f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "78f9ed39-b885-4069-8827-d7441f1ab19f" (UID: "78f9ed39-b885-4069-8827-d7441f1ab19f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:44:48.198713 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.198560 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-data" (OuterVolumeSpecName: "istio-data") pod "78f9ed39-b885-4069-8827-d7441f1ab19f" (UID: "78f9ed39-b885-4069-8827-d7441f1ab19f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:44:48.200202 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.200182 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "78f9ed39-b885-4069-8827-d7441f1ab19f" (UID: "78f9ed39-b885-4069-8827-d7441f1ab19f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:44:48.200292 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.200257 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f9ed39-b885-4069-8827-d7441f1ab19f-kube-api-access-tqh8c" (OuterVolumeSpecName: "kube-api-access-tqh8c") pod "78f9ed39-b885-4069-8827-d7441f1ab19f" (UID: "78f9ed39-b885-4069-8827-d7441f1ab19f"). InnerVolumeSpecName "kube-api-access-tqh8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:44:48.200577 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.200557 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-token" (OuterVolumeSpecName: "istio-token") pod "78f9ed39-b885-4069-8827-d7441f1ab19f" (UID: "78f9ed39-b885-4069-8827-d7441f1ab19f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:44:48.200634 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.200559 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "78f9ed39-b885-4069-8827-d7441f1ab19f" (UID: "78f9ed39-b885-4069-8827-d7441f1ab19f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 14:44:48.299119 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.299084 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-envoy\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:44:48.299119 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.299113 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqh8c\" (UniqueName: \"kubernetes.io/projected/78f9ed39-b885-4069-8827-d7441f1ab19f-kube-api-access-tqh8c\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:44:48.299119 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.299124 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-token\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:44:48.299316 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.299134 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78f9ed39-b885-4069-8827-d7441f1ab19f-istiod-ca-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:44:48.299316 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.299143 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-podinfo\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:44:48.299316 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.299150 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78f9ed39-b885-4069-8827-d7441f1ab19f-istio-data\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:44:48.893416 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.893382 2583 generic.go:356] "Generic (PLEG): container finished" podID="78f9ed39-b885-4069-8827-d7441f1ab19f" containerID="294a1ed3fb35e84fcb0df670967729fbd8f6c3e9f5c4b2bf0d0b2c8d3e3ed7d7" exitCode=0 Jul 07 14:44:48.893562 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.893434 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" Jul 07 14:44:48.893562 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.893472 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" event={"ID":"78f9ed39-b885-4069-8827-d7441f1ab19f","Type":"ContainerDied","Data":"294a1ed3fb35e84fcb0df670967729fbd8f6c3e9f5c4b2bf0d0b2c8d3e3ed7d7"} Jul 07 14:44:48.893562 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.893511 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2" event={"ID":"78f9ed39-b885-4069-8827-d7441f1ab19f","Type":"ContainerDied","Data":"6c45e81db42d44ae99f8797e3cb0e1a28af1f349f74285b47e70f7127deeb332"} Jul 07 14:44:48.893562 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.893529 2583 scope.go:117] "RemoveContainer" containerID="294a1ed3fb35e84fcb0df670967729fbd8f6c3e9f5c4b2bf0d0b2c8d3e3ed7d7" Jul 07 14:44:48.901866 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.901847 2583 scope.go:117] "RemoveContainer" containerID="294a1ed3fb35e84fcb0df670967729fbd8f6c3e9f5c4b2bf0d0b2c8d3e3ed7d7" Jul 07 14:44:48.902120 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:44:48.902099 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"294a1ed3fb35e84fcb0df670967729fbd8f6c3e9f5c4b2bf0d0b2c8d3e3ed7d7\": container with ID starting with 294a1ed3fb35e84fcb0df670967729fbd8f6c3e9f5c4b2bf0d0b2c8d3e3ed7d7 not found: ID does not exist" containerID="294a1ed3fb35e84fcb0df670967729fbd8f6c3e9f5c4b2bf0d0b2c8d3e3ed7d7" Jul 07 14:44:48.902184 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.902127 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"294a1ed3fb35e84fcb0df670967729fbd8f6c3e9f5c4b2bf0d0b2c8d3e3ed7d7"} err="failed to get container status \"294a1ed3fb35e84fcb0df670967729fbd8f6c3e9f5c4b2bf0d0b2c8d3e3ed7d7\": rpc error: code = NotFound desc = could not find container \"294a1ed3fb35e84fcb0df670967729fbd8f6c3e9f5c4b2bf0d0b2c8d3e3ed7d7\": container with ID starting with 294a1ed3fb35e84fcb0df670967729fbd8f6c3e9f5c4b2bf0d0b2c8d3e3ed7d7 not found: ID does not exist" Jul 07 14:44:48.919495 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.919469 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2"] Jul 07 14:44:48.923603 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:48.923579 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-qgbz2"] Jul 07 14:44:49.791118 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:49.791083 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78f9ed39-b885-4069-8827-d7441f1ab19f" path="/var/lib/kubelet/pods/78f9ed39-b885-4069-8827-d7441f1ab19f/volumes" Jul 07 14:44:53.873305 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:44:53.873276 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6d85f8474c-9wnls" Jul 07 14:45:23.145259 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:23.145218 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:45:23.212376 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:23.212329 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:45:24.886469 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:24.886434 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:45:24.948041 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:24.948013 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:45:30.373176 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:30.373137 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:45:30.479606 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:30.479575 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:45:30.560132 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:30.560105 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:45:31.383723 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:31.383693 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:45:31.525037 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:31.525008 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:45:46.324147 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.324108 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h"] Jul 07 14:45:46.324575 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.324402 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f9ed39-b885-4069-8827-d7441f1ab19f" containerName="istio-proxy" Jul 07 14:45:46.324575 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.324413 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f9ed39-b885-4069-8827-d7441f1ab19f" containerName="istio-proxy" Jul 07 14:45:46.324575 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.324470 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="78f9ed39-b885-4069-8827-d7441f1ab19f" containerName="istio-proxy" Jul 07 14:45:46.328577 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.328558 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h" Jul 07 14:45:46.331157 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.331138 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 14:45:46.331259 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.331139 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 14:45:46.331259 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.331169 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ckssf\"" Jul 07 14:45:46.336885 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.336862 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h"] Jul 07 14:45:46.422259 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.422213 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h\" (UID: \"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h" Jul 07 14:45:46.422472 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.422275 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h\" (UID: \"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h" Jul 07 14:45:46.422472 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.422348 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzmqf\" (UniqueName: \"kubernetes.io/projected/7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c-kube-api-access-hzmqf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h\" (UID: \"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h" Jul 07 14:45:46.523004 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.522968 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h\" (UID: \"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h" Jul 07 14:45:46.523157 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.523018 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h\" (UID: \"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h" Jul 07 14:45:46.523157 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.523060 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzmqf\" (UniqueName: \"kubernetes.io/projected/7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c-kube-api-access-hzmqf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h\" (UID: \"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h" Jul 07 14:45:46.523341 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.523318 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h\" (UID: \"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h" Jul 07 14:45:46.523433 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.523412 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h\" (UID: \"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h" Jul 07 14:45:46.533134 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.533108 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzmqf\" (UniqueName: \"kubernetes.io/projected/7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c-kube-api-access-hzmqf\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h\" (UID: \"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h" Jul 07 14:45:46.638102 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.638036 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h" Jul 07 14:45:46.735976 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.735936 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr"] Jul 07 14:45:46.740010 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.739986 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr" Jul 07 14:45:46.750503 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.750481 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr"] Jul 07 14:45:46.766239 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.766219 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h"] Jul 07 14:45:46.768791 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:45:46.768769 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9751ba_a2f8_4c7b_b3f9_28ef1ce1280c.slice/crio-ff702d931ade28bbea5cb564193af5b4787764dbf2dd10edc3fc149ca494e7ba WatchSource:0}: Error finding container ff702d931ade28bbea5cb564193af5b4787764dbf2dd10edc3fc149ca494e7ba: Status 404 returned error can't find the container with id ff702d931ade28bbea5cb564193af5b4787764dbf2dd10edc3fc149ca494e7ba Jul 07 14:45:46.826101 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.826078 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xshv4\" (UniqueName: \"kubernetes.io/projected/d345aba4-566d-4088-a96a-953b6983c9e7-kube-api-access-xshv4\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr\" (UID: \"d345aba4-566d-4088-a96a-953b6983c9e7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr" Jul 07 14:45:46.826207 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.826118 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d345aba4-566d-4088-a96a-953b6983c9e7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr\" (UID: \"d345aba4-566d-4088-a96a-953b6983c9e7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr" Jul 07 14:45:46.826207 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.826140 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d345aba4-566d-4088-a96a-953b6983c9e7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr\" (UID: \"d345aba4-566d-4088-a96a-953b6983c9e7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr" Jul 07 14:45:46.927473 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.927395 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xshv4\" (UniqueName: \"kubernetes.io/projected/d345aba4-566d-4088-a96a-953b6983c9e7-kube-api-access-xshv4\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr\" (UID: \"d345aba4-566d-4088-a96a-953b6983c9e7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr" Jul 07 14:45:46.927473 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.927462 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d345aba4-566d-4088-a96a-953b6983c9e7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr\" (UID: \"d345aba4-566d-4088-a96a-953b6983c9e7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr" Jul 07 14:45:46.927657 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.927490 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d345aba4-566d-4088-a96a-953b6983c9e7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr\" (UID: \"d345aba4-566d-4088-a96a-953b6983c9e7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr" Jul 07 14:45:46.927817 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.927789 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d345aba4-566d-4088-a96a-953b6983c9e7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr\" (UID: \"d345aba4-566d-4088-a96a-953b6983c9e7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr" Jul 07 14:45:46.927893 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.927834 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d345aba4-566d-4088-a96a-953b6983c9e7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr\" (UID: \"d345aba4-566d-4088-a96a-953b6983c9e7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr" Jul 07 14:45:46.936229 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:46.936209 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xshv4\" (UniqueName: \"kubernetes.io/projected/d345aba4-566d-4088-a96a-953b6983c9e7-kube-api-access-xshv4\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr\" (UID: \"d345aba4-566d-4088-a96a-953b6983c9e7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr" Jul 07 14:45:47.050346 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.050314 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr" Jul 07 14:45:47.129268 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.129236 2583 generic.go:356] "Generic (PLEG): container finished" podID="7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c" containerID="7e3c9065267c5afd835ffe7f4f0e33ca64e862217856696f63440d3f8d948c9c" exitCode=0 Jul 07 14:45:47.129490 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.129347 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h" event={"ID":"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c","Type":"ContainerDied","Data":"7e3c9065267c5afd835ffe7f4f0e33ca64e862217856696f63440d3f8d948c9c"} Jul 07 14:45:47.129490 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.129422 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h" event={"ID":"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c","Type":"ContainerStarted","Data":"ff702d931ade28bbea5cb564193af5b4787764dbf2dd10edc3fc149ca494e7ba"} Jul 07 14:45:47.129894 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.129875 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:45:47.181113 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.181040 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr"] Jul 07 14:45:47.184194 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:45:47.184172 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd345aba4_566d_4088_a96a_953b6983c9e7.slice/crio-50c58200ecb9187763483116251fc1bc595aa8d252385b3125935f530403219e WatchSource:0}: Error finding container 50c58200ecb9187763483116251fc1bc595aa8d252385b3125935f530403219e: Status 404 returned error can't find the container with id 50c58200ecb9187763483116251fc1bc595aa8d252385b3125935f530403219e Jul 07 14:45:47.331200 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.331163 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn"] Jul 07 14:45:47.334533 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.334512 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn" Jul 07 14:45:47.343517 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.343495 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn"] Jul 07 14:45:47.431799 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.431698 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee8d0df4-fa26-41ff-ab72-e891764fdd79-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn\" (UID: \"ee8d0df4-fa26-41ff-ab72-e891764fdd79\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn" Jul 07 14:45:47.431799 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.431774 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6zrq\" (UniqueName: \"kubernetes.io/projected/ee8d0df4-fa26-41ff-ab72-e891764fdd79-kube-api-access-k6zrq\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn\" (UID: \"ee8d0df4-fa26-41ff-ab72-e891764fdd79\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn" Jul 07 14:45:47.431997 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.431816 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee8d0df4-fa26-41ff-ab72-e891764fdd79-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn\" (UID: \"ee8d0df4-fa26-41ff-ab72-e891764fdd79\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn" Jul 07 14:45:47.532310 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.532268 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6zrq\" (UniqueName: \"kubernetes.io/projected/ee8d0df4-fa26-41ff-ab72-e891764fdd79-kube-api-access-k6zrq\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn\" (UID: \"ee8d0df4-fa26-41ff-ab72-e891764fdd79\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn" Jul 07 14:45:47.532310 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.532318 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee8d0df4-fa26-41ff-ab72-e891764fdd79-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn\" (UID: \"ee8d0df4-fa26-41ff-ab72-e891764fdd79\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn" Jul 07 14:45:47.532587 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.532396 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee8d0df4-fa26-41ff-ab72-e891764fdd79-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn\" (UID: \"ee8d0df4-fa26-41ff-ab72-e891764fdd79\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn" Jul 07 14:45:47.532705 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.532684 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee8d0df4-fa26-41ff-ab72-e891764fdd79-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn\" (UID: \"ee8d0df4-fa26-41ff-ab72-e891764fdd79\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn" Jul 07 14:45:47.532790 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.532771 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee8d0df4-fa26-41ff-ab72-e891764fdd79-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn\" (UID: \"ee8d0df4-fa26-41ff-ab72-e891764fdd79\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn" Jul 07 14:45:47.545886 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.545864 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6zrq\" (UniqueName: \"kubernetes.io/projected/ee8d0df4-fa26-41ff-ab72-e891764fdd79-kube-api-access-k6zrq\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn\" (UID: \"ee8d0df4-fa26-41ff-ab72-e891764fdd79\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn" Jul 07 14:45:47.643003 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.642971 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn" Jul 07 14:45:47.771903 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.771877 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn"] Jul 07 14:45:47.773282 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:45:47.773255 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8d0df4_fa26_41ff_ab72_e891764fdd79.slice/crio-d70ab7f6947689a4e2f68d96c74109a7182bff19bdd2d6994012d08eb51155a1 WatchSource:0}: Error finding container d70ab7f6947689a4e2f68d96c74109a7182bff19bdd2d6994012d08eb51155a1: Status 404 returned error can't find the container with id d70ab7f6947689a4e2f68d96c74109a7182bff19bdd2d6994012d08eb51155a1 Jul 07 14:45:47.924332 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.924300 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh"] Jul 07 14:45:47.927504 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.927484 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh" Jul 07 14:45:47.936414 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:47.936392 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh"] Jul 07 14:45:48.036413 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:48.036383 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d209042-711f-4e4a-9ccc-c1c3d4963546-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh\" (UID: \"3d209042-711f-4e4a-9ccc-c1c3d4963546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh" Jul 07 14:45:48.036564 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:48.036429 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mrs9\" (UniqueName: \"kubernetes.io/projected/3d209042-711f-4e4a-9ccc-c1c3d4963546-kube-api-access-6mrs9\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh\" (UID: \"3d209042-711f-4e4a-9ccc-c1c3d4963546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh" Jul 07 14:45:48.036564 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:48.036517 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d209042-711f-4e4a-9ccc-c1c3d4963546-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh\" (UID: \"3d209042-711f-4e4a-9ccc-c1c3d4963546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh" Jul 07 14:45:48.134415 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:48.134385 2583 generic.go:356] "Generic (PLEG): container finished" podID="ee8d0df4-fa26-41ff-ab72-e891764fdd79" containerID="f7a53d1715af48972469a8cc8e0a7d25594e9111747ffb29db708c1b868ed18e" exitCode=0 Jul 07 14:45:48.134569 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:48.134449 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn" event={"ID":"ee8d0df4-fa26-41ff-ab72-e891764fdd79","Type":"ContainerDied","Data":"f7a53d1715af48972469a8cc8e0a7d25594e9111747ffb29db708c1b868ed18e"} Jul 07 14:45:48.134569 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:48.134531 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn" event={"ID":"ee8d0df4-fa26-41ff-ab72-e891764fdd79","Type":"ContainerStarted","Data":"d70ab7f6947689a4e2f68d96c74109a7182bff19bdd2d6994012d08eb51155a1"} Jul 07 14:45:48.136022 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:48.135995 2583 generic.go:356] "Generic (PLEG): container finished" podID="d345aba4-566d-4088-a96a-953b6983c9e7" containerID="bcdb4e26b1e1e9e4e21cef419c5138a4bbf40494c8c42507af0bfd23ac3cac30" exitCode=0 Jul 07 14:45:48.136107 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:48.136068 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr" event={"ID":"d345aba4-566d-4088-a96a-953b6983c9e7","Type":"ContainerDied","Data":"bcdb4e26b1e1e9e4e21cef419c5138a4bbf40494c8c42507af0bfd23ac3cac30"} Jul 07 14:45:48.136107 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:48.136104 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr" event={"ID":"d345aba4-566d-4088-a96a-953b6983c9e7","Type":"ContainerStarted","Data":"50c58200ecb9187763483116251fc1bc595aa8d252385b3125935f530403219e"} Jul 07 14:45:48.137585 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:48.136930 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d209042-711f-4e4a-9ccc-c1c3d4963546-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh\" (UID: \"3d209042-711f-4e4a-9ccc-c1c3d4963546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh" Jul 07 14:45:48.137585 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:48.137008 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d209042-711f-4e4a-9ccc-c1c3d4963546-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh\" (UID: \"3d209042-711f-4e4a-9ccc-c1c3d4963546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh" Jul 07 14:45:48.137585 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:48.137067 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mrs9\" (UniqueName: \"kubernetes.io/projected/3d209042-711f-4e4a-9ccc-c1c3d4963546-kube-api-access-6mrs9\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh\" (UID: \"3d209042-711f-4e4a-9ccc-c1c3d4963546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh" Jul 07 14:45:48.139996 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:48.139978 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d209042-711f-4e4a-9ccc-c1c3d4963546-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh\" (UID: \"3d209042-711f-4e4a-9ccc-c1c3d4963546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh" Jul 07 14:45:48.140094 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:48.140055 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d209042-711f-4e4a-9ccc-c1c3d4963546-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh\" (UID: \"3d209042-711f-4e4a-9ccc-c1c3d4963546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh" Jul 07 14:45:48.146862 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:48.146839 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mrs9\" (UniqueName: \"kubernetes.io/projected/3d209042-711f-4e4a-9ccc-c1c3d4963546-kube-api-access-6mrs9\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh\" (UID: \"3d209042-711f-4e4a-9ccc-c1c3d4963546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh" Jul 07 14:45:48.235988 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:48.235966 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh" Jul 07 14:45:48.364165 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:48.364138 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh"] Jul 07 14:45:48.365743 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:45:48.365714 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d209042_711f_4e4a_9ccc_c1c3d4963546.slice/crio-f78ffa8aa7bcc7138aefce023f8ccc4f56a750ae12d2bbc5ddd4bc08f3774205 WatchSource:0}: Error finding container f78ffa8aa7bcc7138aefce023f8ccc4f56a750ae12d2bbc5ddd4bc08f3774205: Status 404 returned error can't find the container with id f78ffa8aa7bcc7138aefce023f8ccc4f56a750ae12d2bbc5ddd4bc08f3774205 Jul 07 14:45:49.057157 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:45:49.057123 2583 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd345aba4_566d_4088_a96a_953b6983c9e7.slice/crio-conmon-1b7ca76c7c2cb8d0eaf32625813f667a4edc4147672d4e41ffd230f98cafbffc.scope\": RecentStats: unable to find data in memory cache]" Jul 07 14:45:49.142182 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:49.142147 2583 generic.go:356] "Generic (PLEG): container finished" podID="7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c" containerID="061021cac982e0950fdf109a2e9611c78808ff8d862623919fb473da85f5b75b" exitCode=0 Jul 07 14:45:49.142302 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:49.142237 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h" event={"ID":"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c","Type":"ContainerDied","Data":"061021cac982e0950fdf109a2e9611c78808ff8d862623919fb473da85f5b75b"} Jul 07 14:45:49.143848 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:49.143822 2583 generic.go:356] "Generic (PLEG): container finished" podID="3d209042-711f-4e4a-9ccc-c1c3d4963546" containerID="e11e225134452b3b7311e2f4e1373aaa31c0113174e645ee9eb7ec674f95175a" exitCode=0 Jul 07 14:45:49.143947 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:49.143926 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh" event={"ID":"3d209042-711f-4e4a-9ccc-c1c3d4963546","Type":"ContainerDied","Data":"e11e225134452b3b7311e2f4e1373aaa31c0113174e645ee9eb7ec674f95175a"} Jul 07 14:45:49.144052 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:49.143969 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh" event={"ID":"3d209042-711f-4e4a-9ccc-c1c3d4963546","Type":"ContainerStarted","Data":"f78ffa8aa7bcc7138aefce023f8ccc4f56a750ae12d2bbc5ddd4bc08f3774205"} Jul 07 14:45:49.145743 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:49.145719 2583 generic.go:356] "Generic (PLEG): container finished" podID="ee8d0df4-fa26-41ff-ab72-e891764fdd79" containerID="3ad27109d3b5b369642b2635f84fd51ca59a48d0b1c457f62994af9c39401fd9" exitCode=0 Jul 07 14:45:49.145829 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:49.145802 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn" event={"ID":"ee8d0df4-fa26-41ff-ab72-e891764fdd79","Type":"ContainerDied","Data":"3ad27109d3b5b369642b2635f84fd51ca59a48d0b1c457f62994af9c39401fd9"} Jul 07 14:45:49.147839 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:49.147819 2583 generic.go:356] "Generic (PLEG): container finished" podID="d345aba4-566d-4088-a96a-953b6983c9e7" containerID="1b7ca76c7c2cb8d0eaf32625813f667a4edc4147672d4e41ffd230f98cafbffc" exitCode=0 Jul 07 14:45:49.147927 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:49.147893 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr" event={"ID":"d345aba4-566d-4088-a96a-953b6983c9e7","Type":"ContainerDied","Data":"1b7ca76c7c2cb8d0eaf32625813f667a4edc4147672d4e41ffd230f98cafbffc"} Jul 07 14:45:50.154245 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:50.154164 2583 generic.go:356] "Generic (PLEG): container finished" podID="3d209042-711f-4e4a-9ccc-c1c3d4963546" containerID="bde92a628b558c421427d86d108d77ea861624430f4377d91011957a91f6638d" exitCode=0 Jul 07 14:45:50.154706 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:50.154251 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh" event={"ID":"3d209042-711f-4e4a-9ccc-c1c3d4963546","Type":"ContainerDied","Data":"bde92a628b558c421427d86d108d77ea861624430f4377d91011957a91f6638d"} Jul 07 14:45:50.156922 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:50.156892 2583 generic.go:356] "Generic (PLEG): container finished" podID="ee8d0df4-fa26-41ff-ab72-e891764fdd79" containerID="86f3940f6cc1839d303cad885e538f0eb3e29c713033090ad711cfd088336500" exitCode=0 Jul 07 14:45:50.160195 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:50.157033 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn" event={"ID":"ee8d0df4-fa26-41ff-ab72-e891764fdd79","Type":"ContainerDied","Data":"86f3940f6cc1839d303cad885e538f0eb3e29c713033090ad711cfd088336500"} Jul 07 14:45:50.163715 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:50.163694 2583 generic.go:356] "Generic (PLEG): container finished" podID="d345aba4-566d-4088-a96a-953b6983c9e7" containerID="3f3b25c942bed47cd7fa0f2f3af3141fb436a0e8b339d6254cb63a7ef1246efc" exitCode=0 Jul 07 14:45:50.163814 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:50.163778 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr" event={"ID":"d345aba4-566d-4088-a96a-953b6983c9e7","Type":"ContainerDied","Data":"3f3b25c942bed47cd7fa0f2f3af3141fb436a0e8b339d6254cb63a7ef1246efc"} Jul 07 14:45:50.165949 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:50.165924 2583 generic.go:356] "Generic (PLEG): container finished" podID="7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c" containerID="7b5e5ca39a235d6a6a1811be8aecbdfed5470ba8d6946cb6d03ec50937aac4bf" exitCode=0 Jul 07 14:45:50.166047 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:50.165948 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h" event={"ID":"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c","Type":"ContainerDied","Data":"7b5e5ca39a235d6a6a1811be8aecbdfed5470ba8d6946cb6d03ec50937aac4bf"} Jul 07 14:45:51.171832 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.171789 2583 generic.go:356] "Generic (PLEG): container finished" podID="3d209042-711f-4e4a-9ccc-c1c3d4963546" containerID="201b3c51be99f2398721f4e3ac97bd9e3436b8b1da303b425404ca42b6468b08" exitCode=0 Jul 07 14:45:51.172253 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.171875 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh" event={"ID":"3d209042-711f-4e4a-9ccc-c1c3d4963546","Type":"ContainerDied","Data":"201b3c51be99f2398721f4e3ac97bd9e3436b8b1da303b425404ca42b6468b08"} Jul 07 14:45:51.313152 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.313123 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr" Jul 07 14:45:51.375344 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.375319 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h" Jul 07 14:45:51.378022 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.378005 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn" Jul 07 14:45:51.464175 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.464102 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d345aba4-566d-4088-a96a-953b6983c9e7-bundle\") pod \"d345aba4-566d-4088-a96a-953b6983c9e7\" (UID: \"d345aba4-566d-4088-a96a-953b6983c9e7\") " Jul 07 14:45:51.464300 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.464172 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d345aba4-566d-4088-a96a-953b6983c9e7-util\") pod \"d345aba4-566d-4088-a96a-953b6983c9e7\" (UID: \"d345aba4-566d-4088-a96a-953b6983c9e7\") " Jul 07 14:45:51.464300 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.464215 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xshv4\" (UniqueName: \"kubernetes.io/projected/d345aba4-566d-4088-a96a-953b6983c9e7-kube-api-access-xshv4\") pod \"d345aba4-566d-4088-a96a-953b6983c9e7\" (UID: \"d345aba4-566d-4088-a96a-953b6983c9e7\") " Jul 07 14:45:51.464670 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.464641 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d345aba4-566d-4088-a96a-953b6983c9e7-bundle" (OuterVolumeSpecName: "bundle") pod "d345aba4-566d-4088-a96a-953b6983c9e7" (UID: "d345aba4-566d-4088-a96a-953b6983c9e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:45:51.466439 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.466409 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d345aba4-566d-4088-a96a-953b6983c9e7-kube-api-access-xshv4" (OuterVolumeSpecName: "kube-api-access-xshv4") pod "d345aba4-566d-4088-a96a-953b6983c9e7" (UID: "d345aba4-566d-4088-a96a-953b6983c9e7"). InnerVolumeSpecName "kube-api-access-xshv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:45:51.469724 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.469698 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d345aba4-566d-4088-a96a-953b6983c9e7-util" (OuterVolumeSpecName: "util") pod "d345aba4-566d-4088-a96a-953b6983c9e7" (UID: "d345aba4-566d-4088-a96a-953b6983c9e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:45:51.564770 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.564736 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee8d0df4-fa26-41ff-ab72-e891764fdd79-bundle\") pod \"ee8d0df4-fa26-41ff-ab72-e891764fdd79\" (UID: \"ee8d0df4-fa26-41ff-ab72-e891764fdd79\") " Jul 07 14:45:51.564846 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.564808 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c-util\") pod \"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c\" (UID: \"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c\") " Jul 07 14:45:51.564846 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.564823 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c-bundle\") pod \"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c\" (UID: \"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c\") " Jul 07 14:45:51.564846 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.564840 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6zrq\" (UniqueName: \"kubernetes.io/projected/ee8d0df4-fa26-41ff-ab72-e891764fdd79-kube-api-access-k6zrq\") pod \"ee8d0df4-fa26-41ff-ab72-e891764fdd79\" (UID: \"ee8d0df4-fa26-41ff-ab72-e891764fdd79\") " Jul 07 14:45:51.564986 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.564858 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzmqf\" (UniqueName: \"kubernetes.io/projected/7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c-kube-api-access-hzmqf\") pod \"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c\" (UID: \"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c\") " Jul 07 14:45:51.564986 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.564873 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee8d0df4-fa26-41ff-ab72-e891764fdd79-util\") pod \"ee8d0df4-fa26-41ff-ab72-e891764fdd79\" (UID: \"ee8d0df4-fa26-41ff-ab72-e891764fdd79\") " Jul 07 14:45:51.565087 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.565025 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d345aba4-566d-4088-a96a-953b6983c9e7-bundle\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:45:51.565087 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.565042 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d345aba4-566d-4088-a96a-953b6983c9e7-util\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:45:51.565087 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.565055 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xshv4\" (UniqueName: \"kubernetes.io/projected/d345aba4-566d-4088-a96a-953b6983c9e7-kube-api-access-xshv4\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:45:51.565291 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.565262 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8d0df4-fa26-41ff-ab72-e891764fdd79-bundle" (OuterVolumeSpecName: "bundle") pod "ee8d0df4-fa26-41ff-ab72-e891764fdd79" (UID: "ee8d0df4-fa26-41ff-ab72-e891764fdd79"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:45:51.565404 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.565383 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c-bundle" (OuterVolumeSpecName: "bundle") pod "7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c" (UID: "7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:45:51.566832 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.566808 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c-kube-api-access-hzmqf" (OuterVolumeSpecName: "kube-api-access-hzmqf") pod "7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c" (UID: "7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c"). InnerVolumeSpecName "kube-api-access-hzmqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:45:51.567189 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.567173 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8d0df4-fa26-41ff-ab72-e891764fdd79-kube-api-access-k6zrq" (OuterVolumeSpecName: "kube-api-access-k6zrq") pod "ee8d0df4-fa26-41ff-ab72-e891764fdd79" (UID: "ee8d0df4-fa26-41ff-ab72-e891764fdd79"). InnerVolumeSpecName "kube-api-access-k6zrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:45:51.570835 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.570811 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8d0df4-fa26-41ff-ab72-e891764fdd79-util" (OuterVolumeSpecName: "util") pod "ee8d0df4-fa26-41ff-ab72-e891764fdd79" (UID: "ee8d0df4-fa26-41ff-ab72-e891764fdd79"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:45:51.571022 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.571002 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c-util" (OuterVolumeSpecName: "util") pod "7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c" (UID: "7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:45:51.665610 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.665587 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee8d0df4-fa26-41ff-ab72-e891764fdd79-bundle\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:45:51.665610 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.665611 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c-util\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:45:51.665729 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.665621 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c-bundle\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:45:51.665729 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.665629 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6zrq\" (UniqueName: \"kubernetes.io/projected/ee8d0df4-fa26-41ff-ab72-e891764fdd79-kube-api-access-k6zrq\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:45:51.665729 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.665640 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzmqf\" (UniqueName: \"kubernetes.io/projected/7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c-kube-api-access-hzmqf\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:45:51.665729 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:51.665648 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee8d0df4-fa26-41ff-ab72-e891764fdd79-util\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:45:52.178110 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:52.178025 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn" Jul 07 14:45:52.178495 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:52.178024 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn" event={"ID":"ee8d0df4-fa26-41ff-ab72-e891764fdd79","Type":"ContainerDied","Data":"d70ab7f6947689a4e2f68d96c74109a7182bff19bdd2d6994012d08eb51155a1"} Jul 07 14:45:52.178495 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:52.178172 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d70ab7f6947689a4e2f68d96c74109a7182bff19bdd2d6994012d08eb51155a1" Jul 07 14:45:52.180020 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:52.179929 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr" Jul 07 14:45:52.180020 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:52.179963 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr" event={"ID":"d345aba4-566d-4088-a96a-953b6983c9e7","Type":"ContainerDied","Data":"50c58200ecb9187763483116251fc1bc595aa8d252385b3125935f530403219e"} Jul 07 14:45:52.180020 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:52.180004 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50c58200ecb9187763483116251fc1bc595aa8d252385b3125935f530403219e" Jul 07 14:45:52.182071 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:52.182043 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h" Jul 07 14:45:52.182178 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:52.182040 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h" event={"ID":"7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c","Type":"ContainerDied","Data":"ff702d931ade28bbea5cb564193af5b4787764dbf2dd10edc3fc149ca494e7ba"} Jul 07 14:45:52.182178 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:52.182107 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff702d931ade28bbea5cb564193af5b4787764dbf2dd10edc3fc149ca494e7ba" Jul 07 14:45:52.290808 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:52.290785 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh" Jul 07 14:45:52.474018 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:52.473992 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d209042-711f-4e4a-9ccc-c1c3d4963546-bundle\") pod \"3d209042-711f-4e4a-9ccc-c1c3d4963546\" (UID: \"3d209042-711f-4e4a-9ccc-c1c3d4963546\") " Jul 07 14:45:52.474182 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:52.474058 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mrs9\" (UniqueName: \"kubernetes.io/projected/3d209042-711f-4e4a-9ccc-c1c3d4963546-kube-api-access-6mrs9\") pod \"3d209042-711f-4e4a-9ccc-c1c3d4963546\" (UID: \"3d209042-711f-4e4a-9ccc-c1c3d4963546\") " Jul 07 14:45:52.474182 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:52.474084 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d209042-711f-4e4a-9ccc-c1c3d4963546-util\") pod \"3d209042-711f-4e4a-9ccc-c1c3d4963546\" (UID: \"3d209042-711f-4e4a-9ccc-c1c3d4963546\") " Jul 07 14:45:52.474713 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:52.474691 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d209042-711f-4e4a-9ccc-c1c3d4963546-bundle" (OuterVolumeSpecName: "bundle") pod "3d209042-711f-4e4a-9ccc-c1c3d4963546" (UID: "3d209042-711f-4e4a-9ccc-c1c3d4963546"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:45:52.476159 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:52.476136 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d209042-711f-4e4a-9ccc-c1c3d4963546-kube-api-access-6mrs9" (OuterVolumeSpecName: "kube-api-access-6mrs9") pod "3d209042-711f-4e4a-9ccc-c1c3d4963546" (UID: "3d209042-711f-4e4a-9ccc-c1c3d4963546"). InnerVolumeSpecName "kube-api-access-6mrs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:45:52.479569 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:52.479548 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d209042-711f-4e4a-9ccc-c1c3d4963546-util" (OuterVolumeSpecName: "util") pod "3d209042-711f-4e4a-9ccc-c1c3d4963546" (UID: "3d209042-711f-4e4a-9ccc-c1c3d4963546"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:45:52.574927 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:52.574904 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d209042-711f-4e4a-9ccc-c1c3d4963546-bundle\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:45:52.574927 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:52.574925 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mrs9\" (UniqueName: \"kubernetes.io/projected/3d209042-711f-4e4a-9ccc-c1c3d4963546-kube-api-access-6mrs9\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:45:52.575096 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:52.574935 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d209042-711f-4e4a-9ccc-c1c3d4963546-util\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:45:53.188421 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:53.188386 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh" event={"ID":"3d209042-711f-4e4a-9ccc-c1c3d4963546","Type":"ContainerDied","Data":"f78ffa8aa7bcc7138aefce023f8ccc4f56a750ae12d2bbc5ddd4bc08f3774205"} Jul 07 14:45:53.188812 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:53.188432 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f78ffa8aa7bcc7138aefce023f8ccc4f56a750ae12d2bbc5ddd4bc08f3774205" Jul 07 14:45:53.188812 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:53.188402 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh" Jul 07 14:45:58.312591 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:58.312558 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:45:59.299962 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:59.299927 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:45:59.903496 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:45:59.903461 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:00.945663 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:00.945632 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:04.551863 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:04.551829 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:04.717070 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:04.717043 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:06.728941 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:06.728907 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:07.028104 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:07.028020 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:07.968948 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:07.968912 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:10.786152 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:10.786113 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:11.775822 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:11.775777 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:13.716083 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:13.716050 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:13.808555 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:13.808524 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:14.116798 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:14.116762 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:14.175366 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:14.175338 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:17.915670 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:17.915637 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:18.023190 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:18.023152 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:19.573404 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:19.573304 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:19.673853 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:19.673822 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:23.547082 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:23.547050 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:23.613616 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:23.613594 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:29.705091 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:29.705053 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:29.854993 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:29.854965 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:33.567978 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:33.567938 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:33.668715 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:33.668678 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:34.166349 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:34.166321 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:34.268325 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:34.268291 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:37.176215 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:37.176181 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:37.246027 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:37.245998 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:41.638340 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:41.638307 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:41.710960 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:41.710932 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:48.389535 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:48.389494 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:48.492865 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:48.492836 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:59.683246 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:59.683205 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:59.854785 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:46:59.854756 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:12.212769 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.212738 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7"] Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.212998 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee8d0df4-fa26-41ff-ab72-e891764fdd79" containerName="util" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213007 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee8d0df4-fa26-41ff-ab72-e891764fdd79" containerName="util" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213015 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d209042-711f-4e4a-9ccc-c1c3d4963546" containerName="util" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213021 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d209042-711f-4e4a-9ccc-c1c3d4963546" containerName="util" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213029 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee8d0df4-fa26-41ff-ab72-e891764fdd79" containerName="pull" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213034 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee8d0df4-fa26-41ff-ab72-e891764fdd79" containerName="pull" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213040 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee8d0df4-fa26-41ff-ab72-e891764fdd79" containerName="extract" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213045 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee8d0df4-fa26-41ff-ab72-e891764fdd79" containerName="extract" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213054 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d209042-711f-4e4a-9ccc-c1c3d4963546" containerName="pull" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213059 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d209042-711f-4e4a-9ccc-c1c3d4963546" containerName="pull" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213066 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c" containerName="pull" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213071 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c" containerName="pull" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213076 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d345aba4-566d-4088-a96a-953b6983c9e7" containerName="extract" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213082 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d345aba4-566d-4088-a96a-953b6983c9e7" containerName="extract" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213093 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d345aba4-566d-4088-a96a-953b6983c9e7" containerName="pull" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213098 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d345aba4-566d-4088-a96a-953b6983c9e7" containerName="pull" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213103 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c" containerName="extract" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213109 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c" containerName="extract" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213115 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d345aba4-566d-4088-a96a-953b6983c9e7" containerName="util" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213121 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d345aba4-566d-4088-a96a-953b6983c9e7" containerName="util" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213127 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d209042-711f-4e4a-9ccc-c1c3d4963546" containerName="extract" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213132 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d209042-711f-4e4a-9ccc-c1c3d4963546" containerName="extract" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213141 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c" containerName="util" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213146 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c" containerName="util" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213188 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d209042-711f-4e4a-9ccc-c1c3d4963546" containerName="extract" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213197 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee8d0df4-fa26-41ff-ab72-e891764fdd79" containerName="extract" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213204 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d345aba4-566d-4088-a96a-953b6983c9e7" containerName="extract" Jul 07 14:47:12.213266 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.213209 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c" containerName="extract" Jul 07 14:47:12.215960 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.215940 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.218506 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.218485 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 14:47:12.218626 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.218508 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gf486\"" Jul 07 14:47:12.230506 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.230484 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7"] Jul 07 14:47:12.255869 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.255844 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:12.317393 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.317348 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.317503 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.317396 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.317503 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.317473 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.317620 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.317501 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.317620 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.317522 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8r4x\" (UniqueName: \"kubernetes.io/projected/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-kube-api-access-c8r4x\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.317620 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.317556 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.317620 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.317578 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.317620 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.317604 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.317874 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.317625 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.418608 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.418571 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.418608 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.418610 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.418812 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.418629 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.418812 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.418650 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.418812 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.418674 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.418812 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.418691 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.418812 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.418726 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.418812 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.418747 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.418812 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.418768 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8r4x\" (UniqueName: \"kubernetes.io/projected/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-kube-api-access-c8r4x\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.419215 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.419039 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.419215 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.419118 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.419215 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.419149 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.419382 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.419220 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.419500 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.419482 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.421127 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.421102 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.421281 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.421264 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.427246 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.427223 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.427688 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.427664 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8r4x\" (UniqueName: \"kubernetes.io/projected/ffb40f68-9d64-4f10-ad88-68cfb5ec23ba-kube-api-access-c8r4x\") pod \"maas-default-gateway-openshift-default-5896d4dd49-rsgc7\" (UID: \"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.525954 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.525929 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:12.617647 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.617612 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:12.653509 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.653479 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7"] Jul 07 14:47:12.656066 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:47:12.656039 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb40f68_9d64_4f10_ad88_68cfb5ec23ba.slice/crio-605cc4d6365b26c969d227bb8783f0e608d03b9c6f4209b5b9fe8f165632e2dd WatchSource:0}: Error finding container 605cc4d6365b26c969d227bb8783f0e608d03b9c6f4209b5b9fe8f165632e2dd: Status 404 returned error can't find the container with id 605cc4d6365b26c969d227bb8783f0e608d03b9c6f4209b5b9fe8f165632e2dd Jul 07 14:47:12.658530 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.658502 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 14:47:12.658606 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.658567 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 14:47:12.658606 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:12.658601 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 14:47:13.538929 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:13.538895 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" event={"ID":"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba","Type":"ContainerStarted","Data":"37c7de5028aaaacd6b499abfd734fe3b33c55bad8f80bc436cbc832557046cdc"} Jul 07 14:47:13.539419 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:13.538948 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" event={"ID":"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba","Type":"ContainerStarted","Data":"605cc4d6365b26c969d227bb8783f0e608d03b9c6f4209b5b9fe8f165632e2dd"} Jul 07 14:47:13.561520 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:13.561475 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" podStartSLOduration=1.5614618660000001 podStartE2EDuration="1.561461866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:47:13.560766115 +0000 UTC m=+766.471959794" watchObservedRunningTime="2026-07-07 14:47:13.561461866 +0000 UTC m=+766.472655544" Jul 07 14:47:14.526532 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:14.526498 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:14.531030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:14.531006 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:14.543738 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:14.543715 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:14.544412 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:14.544395 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:47:28.290537 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:28.290499 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:28.390875 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:28.390826 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:28.452480 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:28.452446 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:29.078270 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.078234 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-8v2xw"] Jul 07 14:47:29.081160 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.081144 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8v2xw" Jul 07 14:47:29.083696 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.083669 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 14:47:29.083835 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.083685 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 14:47:29.084243 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.084226 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kwnz2\"" Jul 07 14:47:29.089142 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.089125 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8v2xw"] Jul 07 14:47:29.118466 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.118437 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:29.212283 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.212249 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:29.250571 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.250534 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54tms\" (UniqueName: \"kubernetes.io/projected/5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49-kube-api-access-54tms\") pod \"authorino-58cd955675-8v2xw\" (UID: \"5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49\") " pod="kuadrant-system/authorino-58cd955675-8v2xw" Jul 07 14:47:29.276308 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.276282 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-sl9ss"] Jul 07 14:47:29.279281 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.279265 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-sl9ss" Jul 07 14:47:29.287196 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.287170 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-sl9ss"] Jul 07 14:47:29.351557 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.351471 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54tms\" (UniqueName: \"kubernetes.io/projected/5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49-kube-api-access-54tms\") pod \"authorino-58cd955675-8v2xw\" (UID: \"5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49\") " pod="kuadrant-system/authorino-58cd955675-8v2xw" Jul 07 14:47:29.351557 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.351517 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xss7t\" (UniqueName: \"kubernetes.io/projected/37e8cf19-c5d3-4e3c-a1ca-3a71220a7558-kube-api-access-xss7t\") pod \"authorino-75d556bd77-sl9ss\" (UID: \"37e8cf19-c5d3-4e3c-a1ca-3a71220a7558\") " pod="kuadrant-system/authorino-75d556bd77-sl9ss" Jul 07 14:47:29.359158 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.359136 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54tms\" (UniqueName: \"kubernetes.io/projected/5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49-kube-api-access-54tms\") pod \"authorino-58cd955675-8v2xw\" (UID: \"5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49\") " pod="kuadrant-system/authorino-58cd955675-8v2xw" Jul 07 14:47:29.390966 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.390943 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8v2xw" Jul 07 14:47:29.452397 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.452336 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xss7t\" (UniqueName: \"kubernetes.io/projected/37e8cf19-c5d3-4e3c-a1ca-3a71220a7558-kube-api-access-xss7t\") pod \"authorino-75d556bd77-sl9ss\" (UID: \"37e8cf19-c5d3-4e3c-a1ca-3a71220a7558\") " pod="kuadrant-system/authorino-75d556bd77-sl9ss" Jul 07 14:47:29.461715 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.461642 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xss7t\" (UniqueName: \"kubernetes.io/projected/37e8cf19-c5d3-4e3c-a1ca-3a71220a7558-kube-api-access-xss7t\") pod \"authorino-75d556bd77-sl9ss\" (UID: \"37e8cf19-c5d3-4e3c-a1ca-3a71220a7558\") " pod="kuadrant-system/authorino-75d556bd77-sl9ss" Jul 07 14:47:29.520154 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.520127 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8v2xw"] Jul 07 14:47:29.522240 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:47:29.522211 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5af3ec52_4ce3_4c66_b22e_39dc9ecf2a49.slice/crio-d2e7accd821824aef9c18151ad541837302cc979a20238294b03123749a92182 WatchSource:0}: Error finding container d2e7accd821824aef9c18151ad541837302cc979a20238294b03123749a92182: Status 404 returned error can't find the container with id d2e7accd821824aef9c18151ad541837302cc979a20238294b03123749a92182 Jul 07 14:47:29.589944 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.589911 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-sl9ss" Jul 07 14:47:29.614006 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.613972 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8v2xw" event={"ID":"5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49","Type":"ContainerStarted","Data":"d2e7accd821824aef9c18151ad541837302cc979a20238294b03123749a92182"} Jul 07 14:47:29.716520 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:29.716477 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-sl9ss"] Jul 07 14:47:29.718560 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:47:29.718532 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37e8cf19_c5d3_4e3c_a1ca_3a71220a7558.slice/crio-249f51afd6b7fa11d0716cbac6a28e96e12620bd8290861bf9e7871008498273 WatchSource:0}: Error finding container 249f51afd6b7fa11d0716cbac6a28e96e12620bd8290861bf9e7871008498273: Status 404 returned error can't find the container with id 249f51afd6b7fa11d0716cbac6a28e96e12620bd8290861bf9e7871008498273 Jul 07 14:47:30.624403 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:30.624350 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-sl9ss" event={"ID":"37e8cf19-c5d3-4e3c-a1ca-3a71220a7558","Type":"ContainerStarted","Data":"249f51afd6b7fa11d0716cbac6a28e96e12620bd8290861bf9e7871008498273"} Jul 07 14:47:33.639774 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:33.639678 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-sl9ss" event={"ID":"37e8cf19-c5d3-4e3c-a1ca-3a71220a7558","Type":"ContainerStarted","Data":"388880756d9241a134afab168d5af1ea40568a4b2dff5799e75bb6546205d57b"} Jul 07 14:47:33.641298 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:33.641273 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8v2xw" event={"ID":"5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49","Type":"ContainerStarted","Data":"a4fb8f843f92f42c0676a36852028ac7cb54b76a13631b87f09632f2a93ed308"} Jul 07 14:47:33.654286 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:33.654237 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-sl9ss" podStartSLOduration=1.12212576 podStartE2EDuration="4.654223049s" totalImagesPullingTime="3.532097289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:47:33.653216927 +0000 UTC m=+786.564410605" watchObservedRunningTime="2026-07-07 14:47:33.654223049 +0000 UTC m=+786.565416781" Jul 07 14:47:33.670259 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:33.670205 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-8v2xw" podStartSLOduration=0.952694942 podStartE2EDuration="4.670186197s" totalImagesPullingTime="3.717491255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:47:33.66980612 +0000 UTC m=+786.580999799" watchObservedRunningTime="2026-07-07 14:47:33.670186197 +0000 UTC m=+786.581379879" Jul 07 14:47:33.701408 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:33.701354 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8v2xw"] Jul 07 14:47:35.649908 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:35.649849 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-8v2xw" podUID="5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49" containerName="authorino" containerID="cri-o://a4fb8f843f92f42c0676a36852028ac7cb54b76a13631b87f09632f2a93ed308" gracePeriod=30 Jul 07 14:47:35.889181 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:35.889157 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8v2xw" Jul 07 14:47:36.007197 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:36.007162 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54tms\" (UniqueName: \"kubernetes.io/projected/5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49-kube-api-access-54tms\") pod \"5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49\" (UID: \"5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49\") " Jul 07 14:47:36.009066 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:36.009029 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49-kube-api-access-54tms" (OuterVolumeSpecName: "kube-api-access-54tms") pod "5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49" (UID: "5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49"). InnerVolumeSpecName "kube-api-access-54tms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:47:36.108461 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:36.108425 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54tms\" (UniqueName: \"kubernetes.io/projected/5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49-kube-api-access-54tms\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:47:36.655872 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:36.655832 2583 generic.go:356] "Generic (PLEG): container finished" podID="5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49" containerID="a4fb8f843f92f42c0676a36852028ac7cb54b76a13631b87f09632f2a93ed308" exitCode=0 Jul 07 14:47:36.655872 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:36.655873 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8v2xw" Jul 07 14:47:36.656378 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:36.655924 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8v2xw" event={"ID":"5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49","Type":"ContainerDied","Data":"a4fb8f843f92f42c0676a36852028ac7cb54b76a13631b87f09632f2a93ed308"} Jul 07 14:47:36.656378 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:36.655974 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8v2xw" event={"ID":"5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49","Type":"ContainerDied","Data":"d2e7accd821824aef9c18151ad541837302cc979a20238294b03123749a92182"} Jul 07 14:47:36.656378 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:36.655998 2583 scope.go:117] "RemoveContainer" containerID="a4fb8f843f92f42c0676a36852028ac7cb54b76a13631b87f09632f2a93ed308" Jul 07 14:47:36.665148 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:36.665129 2583 scope.go:117] "RemoveContainer" containerID="a4fb8f843f92f42c0676a36852028ac7cb54b76a13631b87f09632f2a93ed308" Jul 07 14:47:36.665415 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:47:36.665398 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4fb8f843f92f42c0676a36852028ac7cb54b76a13631b87f09632f2a93ed308\": container with ID starting with a4fb8f843f92f42c0676a36852028ac7cb54b76a13631b87f09632f2a93ed308 not found: ID does not exist" containerID="a4fb8f843f92f42c0676a36852028ac7cb54b76a13631b87f09632f2a93ed308" Jul 07 14:47:36.665476 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:36.665423 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fb8f843f92f42c0676a36852028ac7cb54b76a13631b87f09632f2a93ed308"} err="failed to get container status \"a4fb8f843f92f42c0676a36852028ac7cb54b76a13631b87f09632f2a93ed308\": rpc error: code = NotFound desc = could not find container \"a4fb8f843f92f42c0676a36852028ac7cb54b76a13631b87f09632f2a93ed308\": container with ID starting with a4fb8f843f92f42c0676a36852028ac7cb54b76a13631b87f09632f2a93ed308 not found: ID does not exist" Jul 07 14:47:36.676252 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:36.676230 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8v2xw"] Jul 07 14:47:36.680443 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:36.680424 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-8v2xw"] Jul 07 14:47:37.793442 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:37.793406 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49" path="/var/lib/kubelet/pods/5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49/volumes" Jul 07 14:47:46.463995 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:46.463959 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:58.513303 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:58.513262 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:58.749029 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:58.748999 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ln5hf"] Jul 07 14:47:58.749286 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:58.749275 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49" containerName="authorino" Jul 07 14:47:58.749329 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:58.749288 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49" containerName="authorino" Jul 07 14:47:58.749388 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:58.749341 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5af3ec52-4ce3-4c66-b22e-39dc9ecf2a49" containerName="authorino" Jul 07 14:47:58.753148 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:58.753120 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ln5hf" Jul 07 14:47:58.758762 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:58.758734 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ln5hf"] Jul 07 14:47:58.776835 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:58.776774 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h9qr\" (UniqueName: \"kubernetes.io/projected/d5a8f506-3a1c-4b17-806e-2463c7c3496c-kube-api-access-8h9qr\") pod \"authorino-5d4b67c446-ln5hf\" (UID: \"d5a8f506-3a1c-4b17-806e-2463c7c3496c\") " pod="kuadrant-system/authorino-5d4b67c446-ln5hf" Jul 07 14:47:58.824125 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:58.824099 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:58.877337 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:58.877313 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h9qr\" (UniqueName: \"kubernetes.io/projected/d5a8f506-3a1c-4b17-806e-2463c7c3496c-kube-api-access-8h9qr\") pod \"authorino-5d4b67c446-ln5hf\" (UID: \"d5a8f506-3a1c-4b17-806e-2463c7c3496c\") " pod="kuadrant-system/authorino-5d4b67c446-ln5hf" Jul 07 14:47:58.885564 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:58.885535 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h9qr\" (UniqueName: \"kubernetes.io/projected/d5a8f506-3a1c-4b17-806e-2463c7c3496c-kube-api-access-8h9qr\") pod \"authorino-5d4b67c446-ln5hf\" (UID: \"d5a8f506-3a1c-4b17-806e-2463c7c3496c\") " pod="kuadrant-system/authorino-5d4b67c446-ln5hf" Jul 07 14:47:58.938457 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:58.938433 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ln5hf"] Jul 07 14:47:58.938668 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:58.938656 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ln5hf" Jul 07 14:47:59.065926 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:59.065857 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-657bf6847b-gxh8v"] Jul 07 14:47:59.068696 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:59.068638 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-657bf6847b-gxh8v" Jul 07 14:47:59.069826 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:59.069799 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ln5hf"] Jul 07 14:47:59.071137 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:59.071118 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 07 14:47:59.071526 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:47:59.071504 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5a8f506_3a1c_4b17_806e_2463c7c3496c.slice/crio-66cae63b009d96a9834528d34a5d931a21f12459928992c083eb45c2af62d786 WatchSource:0}: Error finding container 66cae63b009d96a9834528d34a5d931a21f12459928992c083eb45c2af62d786: Status 404 returned error can't find the container with id 66cae63b009d96a9834528d34a5d931a21f12459928992c083eb45c2af62d786 Jul 07 14:47:59.078221 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:59.078196 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtgq9\" (UniqueName: \"kubernetes.io/projected/5ccb1371-d228-4ed6-bc88-a35ce5f6bc40-kube-api-access-rtgq9\") pod \"authorino-657bf6847b-gxh8v\" (UID: \"5ccb1371-d228-4ed6-bc88-a35ce5f6bc40\") " pod="kuadrant-system/authorino-657bf6847b-gxh8v" Jul 07 14:47:59.078318 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:59.078247 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5ccb1371-d228-4ed6-bc88-a35ce5f6bc40-tls-cert\") pod \"authorino-657bf6847b-gxh8v\" (UID: \"5ccb1371-d228-4ed6-bc88-a35ce5f6bc40\") " pod="kuadrant-system/authorino-657bf6847b-gxh8v" Jul 07 14:47:59.078917 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:59.078898 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-657bf6847b-gxh8v"] Jul 07 14:47:59.179231 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:59.179206 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtgq9\" (UniqueName: \"kubernetes.io/projected/5ccb1371-d228-4ed6-bc88-a35ce5f6bc40-kube-api-access-rtgq9\") pod \"authorino-657bf6847b-gxh8v\" (UID: \"5ccb1371-d228-4ed6-bc88-a35ce5f6bc40\") " pod="kuadrant-system/authorino-657bf6847b-gxh8v" Jul 07 14:47:59.179329 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:59.179241 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5ccb1371-d228-4ed6-bc88-a35ce5f6bc40-tls-cert\") pod \"authorino-657bf6847b-gxh8v\" (UID: \"5ccb1371-d228-4ed6-bc88-a35ce5f6bc40\") " pod="kuadrant-system/authorino-657bf6847b-gxh8v" Jul 07 14:47:59.181474 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:59.181457 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5ccb1371-d228-4ed6-bc88-a35ce5f6bc40-tls-cert\") pod \"authorino-657bf6847b-gxh8v\" (UID: \"5ccb1371-d228-4ed6-bc88-a35ce5f6bc40\") " pod="kuadrant-system/authorino-657bf6847b-gxh8v" Jul 07 14:47:59.187060 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:59.187042 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtgq9\" (UniqueName: \"kubernetes.io/projected/5ccb1371-d228-4ed6-bc88-a35ce5f6bc40-kube-api-access-rtgq9\") pod \"authorino-657bf6847b-gxh8v\" (UID: \"5ccb1371-d228-4ed6-bc88-a35ce5f6bc40\") " pod="kuadrant-system/authorino-657bf6847b-gxh8v" Jul 07 14:47:59.384754 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:59.384690 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-657bf6847b-gxh8v" Jul 07 14:47:59.513577 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:59.513555 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-657bf6847b-gxh8v"] Jul 07 14:47:59.514897 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:47:59.514873 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ccb1371_d228_4ed6_bc88_a35ce5f6bc40.slice/crio-ae8e79a58db79c7f145054876fad2ec5fe01c86f7c2b9691e01ccf6234bc4c88 WatchSource:0}: Error finding container ae8e79a58db79c7f145054876fad2ec5fe01c86f7c2b9691e01ccf6234bc4c88: Status 404 returned error can't find the container with id ae8e79a58db79c7f145054876fad2ec5fe01c86f7c2b9691e01ccf6234bc4c88 Jul 07 14:47:59.760904 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:59.760871 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ln5hf" event={"ID":"d5a8f506-3a1c-4b17-806e-2463c7c3496c","Type":"ContainerStarted","Data":"b54bd93c8a40990b0ff3b81e515426726adfd863b3ff7613d9cab1fe09de36d7"} Jul 07 14:47:59.761091 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:59.760915 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ln5hf" event={"ID":"d5a8f506-3a1c-4b17-806e-2463c7c3496c","Type":"ContainerStarted","Data":"66cae63b009d96a9834528d34a5d931a21f12459928992c083eb45c2af62d786"} Jul 07 14:47:59.761091 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:59.760959 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-ln5hf" podUID="d5a8f506-3a1c-4b17-806e-2463c7c3496c" containerName="authorino" containerID="cri-o://b54bd93c8a40990b0ff3b81e515426726adfd863b3ff7613d9cab1fe09de36d7" gracePeriod=30 Jul 07 14:47:59.762020 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:59.761995 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-657bf6847b-gxh8v" event={"ID":"5ccb1371-d228-4ed6-bc88-a35ce5f6bc40","Type":"ContainerStarted","Data":"ae8e79a58db79c7f145054876fad2ec5fe01c86f7c2b9691e01ccf6234bc4c88"} Jul 07 14:47:59.776647 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:47:59.776609 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-ln5hf" podStartSLOduration=1.344450929 podStartE2EDuration="1.776597764s" totalImagesPullingTime="432.146835ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:47:59.775477404 +0000 UTC m=+812.686671082" watchObservedRunningTime="2026-07-07 14:47:59.776597764 +0000 UTC m=+812.687791463" Jul 07 14:48:00.009925 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:00.009900 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ln5hf" Jul 07 14:48:00.090701 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:00.090671 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8h9qr\" (UniqueName: \"kubernetes.io/projected/d5a8f506-3a1c-4b17-806e-2463c7c3496c-kube-api-access-8h9qr\") pod \"d5a8f506-3a1c-4b17-806e-2463c7c3496c\" (UID: \"d5a8f506-3a1c-4b17-806e-2463c7c3496c\") " Jul 07 14:48:00.092731 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:00.092707 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a8f506-3a1c-4b17-806e-2463c7c3496c-kube-api-access-8h9qr" (OuterVolumeSpecName: "kube-api-access-8h9qr") pod "d5a8f506-3a1c-4b17-806e-2463c7c3496c" (UID: "d5a8f506-3a1c-4b17-806e-2463c7c3496c"). InnerVolumeSpecName "kube-api-access-8h9qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:48:00.191572 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:00.191546 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h9qr\" (UniqueName: \"kubernetes.io/projected/d5a8f506-3a1c-4b17-806e-2463c7c3496c-kube-api-access-8h9qr\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:48:00.767985 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:00.767944 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-657bf6847b-gxh8v" event={"ID":"5ccb1371-d228-4ed6-bc88-a35ce5f6bc40","Type":"ContainerStarted","Data":"47d2c1b48c312ea811ae3e4f22ca55ad97748566ae94b03bb1677d3793a58bbc"} Jul 07 14:48:00.769063 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:00.769039 2583 generic.go:356] "Generic (PLEG): container finished" podID="d5a8f506-3a1c-4b17-806e-2463c7c3496c" containerID="b54bd93c8a40990b0ff3b81e515426726adfd863b3ff7613d9cab1fe09de36d7" exitCode=0 Jul 07 14:48:00.769189 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:00.769072 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ln5hf" Jul 07 14:48:00.769189 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:00.769108 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ln5hf" event={"ID":"d5a8f506-3a1c-4b17-806e-2463c7c3496c","Type":"ContainerDied","Data":"b54bd93c8a40990b0ff3b81e515426726adfd863b3ff7613d9cab1fe09de36d7"} Jul 07 14:48:00.769189 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:00.769143 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ln5hf" event={"ID":"d5a8f506-3a1c-4b17-806e-2463c7c3496c","Type":"ContainerDied","Data":"66cae63b009d96a9834528d34a5d931a21f12459928992c083eb45c2af62d786"} Jul 07 14:48:00.769189 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:00.769166 2583 scope.go:117] "RemoveContainer" containerID="b54bd93c8a40990b0ff3b81e515426726adfd863b3ff7613d9cab1fe09de36d7" Jul 07 14:48:00.777730 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:00.777713 2583 scope.go:117] "RemoveContainer" containerID="b54bd93c8a40990b0ff3b81e515426726adfd863b3ff7613d9cab1fe09de36d7" Jul 07 14:48:00.777951 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:48:00.777935 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b54bd93c8a40990b0ff3b81e515426726adfd863b3ff7613d9cab1fe09de36d7\": container with ID starting with b54bd93c8a40990b0ff3b81e515426726adfd863b3ff7613d9cab1fe09de36d7 not found: ID does not exist" containerID="b54bd93c8a40990b0ff3b81e515426726adfd863b3ff7613d9cab1fe09de36d7" Jul 07 14:48:00.778000 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:00.777956 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b54bd93c8a40990b0ff3b81e515426726adfd863b3ff7613d9cab1fe09de36d7"} err="failed to get container status \"b54bd93c8a40990b0ff3b81e515426726adfd863b3ff7613d9cab1fe09de36d7\": rpc error: code = NotFound desc = could not find container \"b54bd93c8a40990b0ff3b81e515426726adfd863b3ff7613d9cab1fe09de36d7\": container with ID starting with b54bd93c8a40990b0ff3b81e515426726adfd863b3ff7613d9cab1fe09de36d7 not found: ID does not exist" Jul 07 14:48:00.784409 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:00.784354 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-657bf6847b-gxh8v" podStartSLOduration=1.35778594 podStartE2EDuration="1.784341605s" totalImagesPullingTime="426.555665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:48:00.783382934 +0000 UTC m=+813.694576610" watchObservedRunningTime="2026-07-07 14:48:00.784341605 +0000 UTC m=+813.695535282" Jul 07 14:48:00.799906 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:00.799875 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ln5hf"] Jul 07 14:48:00.803748 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:00.803726 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ln5hf"] Jul 07 14:48:00.813275 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:00.813251 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-sl9ss"] Jul 07 14:48:00.813466 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:00.813443 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-sl9ss" podUID="37e8cf19-c5d3-4e3c-a1ca-3a71220a7558" containerName="authorino" containerID="cri-o://388880756d9241a134afab168d5af1ea40568a4b2dff5799e75bb6546205d57b" gracePeriod=30 Jul 07 14:48:01.057537 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:01.057515 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-sl9ss" Jul 07 14:48:01.098278 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:01.098255 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xss7t\" (UniqueName: \"kubernetes.io/projected/37e8cf19-c5d3-4e3c-a1ca-3a71220a7558-kube-api-access-xss7t\") pod \"37e8cf19-c5d3-4e3c-a1ca-3a71220a7558\" (UID: \"37e8cf19-c5d3-4e3c-a1ca-3a71220a7558\") " Jul 07 14:48:01.100183 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:01.100153 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e8cf19-c5d3-4e3c-a1ca-3a71220a7558-kube-api-access-xss7t" (OuterVolumeSpecName: "kube-api-access-xss7t") pod "37e8cf19-c5d3-4e3c-a1ca-3a71220a7558" (UID: "37e8cf19-c5d3-4e3c-a1ca-3a71220a7558"). InnerVolumeSpecName "kube-api-access-xss7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:48:01.199496 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:01.199471 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xss7t\" (UniqueName: \"kubernetes.io/projected/37e8cf19-c5d3-4e3c-a1ca-3a71220a7558-kube-api-access-xss7t\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:48:01.281301 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:01.281262 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:01.356738 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:01.356656 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:01.775372 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:01.775331 2583 generic.go:356] "Generic (PLEG): container finished" podID="37e8cf19-c5d3-4e3c-a1ca-3a71220a7558" containerID="388880756d9241a134afab168d5af1ea40568a4b2dff5799e75bb6546205d57b" exitCode=0 Jul 07 14:48:01.775833 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:01.775380 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-sl9ss" Jul 07 14:48:01.775833 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:01.775400 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-sl9ss" event={"ID":"37e8cf19-c5d3-4e3c-a1ca-3a71220a7558","Type":"ContainerDied","Data":"388880756d9241a134afab168d5af1ea40568a4b2dff5799e75bb6546205d57b"} Jul 07 14:48:01.775833 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:01.775441 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-sl9ss" event={"ID":"37e8cf19-c5d3-4e3c-a1ca-3a71220a7558","Type":"ContainerDied","Data":"249f51afd6b7fa11d0716cbac6a28e96e12620bd8290861bf9e7871008498273"} Jul 07 14:48:01.775833 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:01.775463 2583 scope.go:117] "RemoveContainer" containerID="388880756d9241a134afab168d5af1ea40568a4b2dff5799e75bb6546205d57b" Jul 07 14:48:01.783680 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:01.783662 2583 scope.go:117] "RemoveContainer" containerID="388880756d9241a134afab168d5af1ea40568a4b2dff5799e75bb6546205d57b" Jul 07 14:48:01.783966 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:48:01.783940 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"388880756d9241a134afab168d5af1ea40568a4b2dff5799e75bb6546205d57b\": container with ID starting with 388880756d9241a134afab168d5af1ea40568a4b2dff5799e75bb6546205d57b not found: ID does not exist" containerID="388880756d9241a134afab168d5af1ea40568a4b2dff5799e75bb6546205d57b" Jul 07 14:48:01.784055 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:01.783977 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"388880756d9241a134afab168d5af1ea40568a4b2dff5799e75bb6546205d57b"} err="failed to get container status \"388880756d9241a134afab168d5af1ea40568a4b2dff5799e75bb6546205d57b\": rpc error: code = NotFound desc = could not find container \"388880756d9241a134afab168d5af1ea40568a4b2dff5799e75bb6546205d57b\": container with ID starting with 388880756d9241a134afab168d5af1ea40568a4b2dff5799e75bb6546205d57b not found: ID does not exist" Jul 07 14:48:01.790701 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:01.790671 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5a8f506-3a1c-4b17-806e-2463c7c3496c" path="/var/lib/kubelet/pods/d5a8f506-3a1c-4b17-806e-2463c7c3496c/volumes" Jul 07 14:48:01.797047 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:01.797023 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-sl9ss"] Jul 07 14:48:01.800886 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:01.800865 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-sl9ss"] Jul 07 14:48:03.791347 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:03.791314 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37e8cf19-c5d3-4e3c-a1ca-3a71220a7558" path="/var/lib/kubelet/pods/37e8cf19-c5d3-4e3c-a1ca-3a71220a7558/volumes" Jul 07 14:48:05.269325 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:05.269292 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:05.368693 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:05.368648 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:05.502706 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:05.502673 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:08.873561 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:08.873523 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:08.995708 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:08.995676 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:39.341895 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:39.341857 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:40.795568 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:40.795525 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:47.986583 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:47.986497 2583 generic.go:356] "Generic (PLEG): container finished" podID="ffb40f68-9d64-4f10-ad88-68cfb5ec23ba" containerID="37c7de5028aaaacd6b499abfd734fe3b33c55bad8f80bc436cbc832557046cdc" exitCode=0 Jul 07 14:48:47.987035 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:47.986571 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" event={"ID":"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba","Type":"ContainerDied","Data":"37c7de5028aaaacd6b499abfd734fe3b33c55bad8f80bc436cbc832557046cdc"} Jul 07 14:48:47.987035 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:47.986966 2583 scope.go:117] "RemoveContainer" containerID="37c7de5028aaaacd6b499abfd734fe3b33c55bad8f80bc436cbc832557046cdc" Jul 07 14:48:47.987949 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:47.987922 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 14:48:47.988039 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:47.987991 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 14:48:47.988039 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:47.988021 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 14:48:48.993420 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:48.993386 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" event={"ID":"ffb40f68-9d64-4f10-ad88-68cfb5ec23ba","Type":"ContainerStarted","Data":"57c90359973bbb367dc0be81984c65b58a50b733e160dc604705923f60fb9484"} Jul 07 14:48:49.448569 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:49.447240 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:49.526823 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:49.526787 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:48:49.527046 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:49.527021 2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-rsgc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Jul 07 14:48:49.527131 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:49.527079 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" podUID="ffb40f68-9d64-4f10-ad88-68cfb5ec23ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Jul 07 14:48:49.550287 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:49.550252 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:50.527067 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:50.527026 2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-rsgc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Jul 07 14:48:50.527572 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:50.527088 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" podUID="ffb40f68-9d64-4f10-ad88-68cfb5ec23ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Jul 07 14:48:51.526942 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:51.526909 2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-rsgc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Jul 07 14:48:51.527120 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:51.526964 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" podUID="ffb40f68-9d64-4f10-ad88-68cfb5ec23ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Jul 07 14:48:52.526504 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:52.526466 2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-rsgc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Jul 07 14:48:52.526678 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:52.526526 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" podUID="ffb40f68-9d64-4f10-ad88-68cfb5ec23ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Jul 07 14:48:53.529564 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:53.529531 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:48:53.529984 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:53.529786 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:48:53.530495 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:53.530478 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-rsgc7" Jul 07 14:48:54.006204 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:48:54.006168 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:04.881555 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:04.881514 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:27.579332 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:27.579256 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 14:49:27.582731 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:27.582708 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 14:49:39.960912 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:39.960865 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh"] Jul 07 14:49:39.961553 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:39.961531 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a8f506-3a1c-4b17-806e-2463c7c3496c" containerName="authorino" Jul 07 14:49:39.961631 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:39.961560 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a8f506-3a1c-4b17-806e-2463c7c3496c" containerName="authorino" Jul 07 14:49:39.961631 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:39.961590 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37e8cf19-c5d3-4e3c-a1ca-3a71220a7558" containerName="authorino" Jul 07 14:49:39.961631 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:39.961598 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="37e8cf19-c5d3-4e3c-a1ca-3a71220a7558" containerName="authorino" Jul 07 14:49:39.961772 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:39.961727 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="37e8cf19-c5d3-4e3c-a1ca-3a71220a7558" containerName="authorino" Jul 07 14:49:39.961772 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:39.961757 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5a8f506-3a1c-4b17-806e-2463c7c3496c" containerName="authorino" Jul 07 14:49:39.965322 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:39.965299 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:39.968456 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:39.968438 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-qh5dn\"" Jul 07 14:49:39.968572 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:39.968455 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 07 14:49:39.968572 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:39.968469 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 14:49:39.968572 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:39.968443 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 14:49:39.973746 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:39.973726 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh"] Jul 07 14:49:39.994252 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:39.994227 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:40.066613 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.066584 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df5s8\" (UniqueName: \"kubernetes.io/projected/08121fd0-34c3-4e39-88ba-c64fda4ce303-kube-api-access-df5s8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh\" (UID: \"08121fd0-34c3-4e39-88ba-c64fda4ce303\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:40.066750 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.066636 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08121fd0-34c3-4e39-88ba-c64fda4ce303-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh\" (UID: \"08121fd0-34c3-4e39-88ba-c64fda4ce303\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:40.066750 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.066699 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08121fd0-34c3-4e39-88ba-c64fda4ce303-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh\" (UID: \"08121fd0-34c3-4e39-88ba-c64fda4ce303\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:40.066836 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.066765 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08121fd0-34c3-4e39-88ba-c64fda4ce303-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh\" (UID: \"08121fd0-34c3-4e39-88ba-c64fda4ce303\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:40.066836 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.066794 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08121fd0-34c3-4e39-88ba-c64fda4ce303-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh\" (UID: \"08121fd0-34c3-4e39-88ba-c64fda4ce303\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:40.066836 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.066828 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08121fd0-34c3-4e39-88ba-c64fda4ce303-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh\" (UID: \"08121fd0-34c3-4e39-88ba-c64fda4ce303\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:40.167532 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.167511 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08121fd0-34c3-4e39-88ba-c64fda4ce303-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh\" (UID: \"08121fd0-34c3-4e39-88ba-c64fda4ce303\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:40.167656 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.167540 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08121fd0-34c3-4e39-88ba-c64fda4ce303-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh\" (UID: \"08121fd0-34c3-4e39-88ba-c64fda4ce303\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:40.167656 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.167565 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08121fd0-34c3-4e39-88ba-c64fda4ce303-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh\" (UID: \"08121fd0-34c3-4e39-88ba-c64fda4ce303\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:40.167656 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.167594 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df5s8\" (UniqueName: \"kubernetes.io/projected/08121fd0-34c3-4e39-88ba-c64fda4ce303-kube-api-access-df5s8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh\" (UID: \"08121fd0-34c3-4e39-88ba-c64fda4ce303\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:40.167656 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.167620 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08121fd0-34c3-4e39-88ba-c64fda4ce303-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh\" (UID: \"08121fd0-34c3-4e39-88ba-c64fda4ce303\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:40.167656 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.167643 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08121fd0-34c3-4e39-88ba-c64fda4ce303-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh\" (UID: \"08121fd0-34c3-4e39-88ba-c64fda4ce303\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:40.168044 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.168023 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08121fd0-34c3-4e39-88ba-c64fda4ce303-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh\" (UID: \"08121fd0-34c3-4e39-88ba-c64fda4ce303\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:40.168097 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.168038 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08121fd0-34c3-4e39-88ba-c64fda4ce303-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh\" (UID: \"08121fd0-34c3-4e39-88ba-c64fda4ce303\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:40.168097 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.168078 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08121fd0-34c3-4e39-88ba-c64fda4ce303-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh\" (UID: \"08121fd0-34c3-4e39-88ba-c64fda4ce303\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:40.169961 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.169942 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08121fd0-34c3-4e39-88ba-c64fda4ce303-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh\" (UID: \"08121fd0-34c3-4e39-88ba-c64fda4ce303\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:40.170177 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.170163 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08121fd0-34c3-4e39-88ba-c64fda4ce303-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh\" (UID: \"08121fd0-34c3-4e39-88ba-c64fda4ce303\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:40.177496 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.177473 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df5s8\" (UniqueName: \"kubernetes.io/projected/08121fd0-34c3-4e39-88ba-c64fda4ce303-kube-api-access-df5s8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh\" (UID: \"08121fd0-34c3-4e39-88ba-c64fda4ce303\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:40.277140 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.277116 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:40.404550 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:40.404523 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh"] Jul 07 14:49:40.407018 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:49:40.406989 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08121fd0_34c3_4e39_88ba_c64fda4ce303.slice/crio-c963a01563bfc8c3297c0f9ddf00ee436ccee23968bcc52baf5a0fb2fe14b482 WatchSource:0}: Error finding container c963a01563bfc8c3297c0f9ddf00ee436ccee23968bcc52baf5a0fb2fe14b482: Status 404 returned error can't find the container with id c963a01563bfc8c3297c0f9ddf00ee436ccee23968bcc52baf5a0fb2fe14b482 Jul 07 14:49:41.235048 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:41.235007 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" event={"ID":"08121fd0-34c3-4e39-88ba-c64fda4ce303","Type":"ContainerStarted","Data":"c963a01563bfc8c3297c0f9ddf00ee436ccee23968bcc52baf5a0fb2fe14b482"} Jul 07 14:49:42.241169 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:42.241129 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" event={"ID":"08121fd0-34c3-4e39-88ba-c64fda4ce303","Type":"ContainerStarted","Data":"114ebe11368fdfd6a836fb9c8d2e48586bb4b66ce51cf4429b9516c6d0b2dc8c"} Jul 07 14:49:42.258479 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:42.258429 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" podStartSLOduration=2.191765969 podStartE2EDuration="3.258410745s" totalImagesPullingTime="1.066644776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:49:42.257328121 +0000 UTC m=+915.168521812" watchObservedRunningTime="2026-07-07 14:49:42.258410745 +0000 UTC m=+915.169604424" Jul 07 14:49:45.282139 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:45.282103 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:50.277228 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:50.277199 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:50.277727 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:50.277374 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:50.289622 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:50.289603 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:51.294209 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:51.294162 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh" Jul 07 14:49:58.863604 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:58.863558 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn"] Jul 07 14:49:58.866888 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:58.866866 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:58.869372 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:58.869345 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 07 14:49:58.880528 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:58.880500 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn"] Jul 07 14:49:58.904663 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:58.904636 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:59.010050 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.010024 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5069baf9-98db-4e4b-a0f8-1a342f61d1f4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn\" (UID: \"5069baf9-98db-4e4b-a0f8-1a342f61d1f4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:59.010171 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.010059 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5069baf9-98db-4e4b-a0f8-1a342f61d1f4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn\" (UID: \"5069baf9-98db-4e4b-a0f8-1a342f61d1f4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:59.010171 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.010094 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5069baf9-98db-4e4b-a0f8-1a342f61d1f4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn\" (UID: \"5069baf9-98db-4e4b-a0f8-1a342f61d1f4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:59.010289 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.010193 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5069baf9-98db-4e4b-a0f8-1a342f61d1f4-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn\" (UID: \"5069baf9-98db-4e4b-a0f8-1a342f61d1f4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:59.010289 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.010233 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5069baf9-98db-4e4b-a0f8-1a342f61d1f4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn\" (UID: \"5069baf9-98db-4e4b-a0f8-1a342f61d1f4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:59.010289 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.010262 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djcs5\" (UniqueName: \"kubernetes.io/projected/5069baf9-98db-4e4b-a0f8-1a342f61d1f4-kube-api-access-djcs5\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn\" (UID: \"5069baf9-98db-4e4b-a0f8-1a342f61d1f4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:59.110972 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.110939 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5069baf9-98db-4e4b-a0f8-1a342f61d1f4-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn\" (UID: \"5069baf9-98db-4e4b-a0f8-1a342f61d1f4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:59.110972 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.110976 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5069baf9-98db-4e4b-a0f8-1a342f61d1f4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn\" (UID: \"5069baf9-98db-4e4b-a0f8-1a342f61d1f4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:59.111169 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.110996 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djcs5\" (UniqueName: \"kubernetes.io/projected/5069baf9-98db-4e4b-a0f8-1a342f61d1f4-kube-api-access-djcs5\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn\" (UID: \"5069baf9-98db-4e4b-a0f8-1a342f61d1f4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:59.111169 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.111022 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5069baf9-98db-4e4b-a0f8-1a342f61d1f4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn\" (UID: \"5069baf9-98db-4e4b-a0f8-1a342f61d1f4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:59.111169 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.111047 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5069baf9-98db-4e4b-a0f8-1a342f61d1f4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn\" (UID: \"5069baf9-98db-4e4b-a0f8-1a342f61d1f4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:59.111169 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.111090 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5069baf9-98db-4e4b-a0f8-1a342f61d1f4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn\" (UID: \"5069baf9-98db-4e4b-a0f8-1a342f61d1f4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:59.111410 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.111350 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5069baf9-98db-4e4b-a0f8-1a342f61d1f4-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn\" (UID: \"5069baf9-98db-4e4b-a0f8-1a342f61d1f4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:59.111515 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.111423 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5069baf9-98db-4e4b-a0f8-1a342f61d1f4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn\" (UID: \"5069baf9-98db-4e4b-a0f8-1a342f61d1f4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:59.111515 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.111485 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5069baf9-98db-4e4b-a0f8-1a342f61d1f4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn\" (UID: \"5069baf9-98db-4e4b-a0f8-1a342f61d1f4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:59.113242 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.113224 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5069baf9-98db-4e4b-a0f8-1a342f61d1f4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn\" (UID: \"5069baf9-98db-4e4b-a0f8-1a342f61d1f4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:59.113528 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.113510 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5069baf9-98db-4e4b-a0f8-1a342f61d1f4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn\" (UID: \"5069baf9-98db-4e4b-a0f8-1a342f61d1f4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:59.119232 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.119180 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djcs5\" (UniqueName: \"kubernetes.io/projected/5069baf9-98db-4e4b-a0f8-1a342f61d1f4-kube-api-access-djcs5\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn\" (UID: \"5069baf9-98db-4e4b-a0f8-1a342f61d1f4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:59.176875 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.176852 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:49:59.311984 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.311947 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn"] Jul 07 14:49:59.314758 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:49:59.314732 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5069baf9_98db_4e4b_a0f8_1a342f61d1f4.slice/crio-7917866f5f4ef16f137dc352281b4d968fe36ce5e9ad04c0cd5ca93f70aff345 WatchSource:0}: Error finding container 7917866f5f4ef16f137dc352281b4d968fe36ce5e9ad04c0cd5ca93f70aff345: Status 404 returned error can't find the container with id 7917866f5f4ef16f137dc352281b4d968fe36ce5e9ad04c0cd5ca93f70aff345 Jul 07 14:49:59.321806 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:49:59.321781 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" event={"ID":"5069baf9-98db-4e4b-a0f8-1a342f61d1f4","Type":"ContainerStarted","Data":"7917866f5f4ef16f137dc352281b4d968fe36ce5e9ad04c0cd5ca93f70aff345"} Jul 07 14:50:00.328387 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:50:00.328337 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" event={"ID":"5069baf9-98db-4e4b-a0f8-1a342f61d1f4","Type":"ContainerStarted","Data":"cb649f57bc44b4106bb893be17f328896f14e1d0f06723fb9ff128a90f5ef6ce"} Jul 07 14:50:00.348910 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:50:00.348862 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" podStartSLOduration=2.152875357 podStartE2EDuration="2.348847458s" totalImagesPullingTime="195.972101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:50:00.347248001 +0000 UTC m=+933.258441679" watchObservedRunningTime="2026-07-07 14:50:00.348847458 +0000 UTC m=+933.260041136" Jul 07 14:50:09.177191 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:50:09.177153 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:50:09.177752 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:50:09.177231 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:50:09.189581 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:50:09.189557 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:50:09.383028 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:50:09.383000 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn" Jul 07 14:50:16.604592 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:50:16.604553 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:16.704934 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:50:16.704892 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:54.189271 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:50:54.189239 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:51:14.425986 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:14.425956 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f5b44b88f-w7cpd"] Jul 07 14:51:14.429325 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:14.429303 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5b44b88f-w7cpd" Jul 07 14:51:14.441958 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:14.441935 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f5b44b88f-w7cpd"] Jul 07 14:51:14.477662 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:14.477640 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/497257b3-dfd7-4c1e-9453-a86f4a626df5-tls-cert\") pod \"authorino-5f5b44b88f-w7cpd\" (UID: \"497257b3-dfd7-4c1e-9453-a86f4a626df5\") " pod="kuadrant-system/authorino-5f5b44b88f-w7cpd" Jul 07 14:51:14.477791 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:14.477696 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b98vk\" (UniqueName: \"kubernetes.io/projected/497257b3-dfd7-4c1e-9453-a86f4a626df5-kube-api-access-b98vk\") pod \"authorino-5f5b44b88f-w7cpd\" (UID: \"497257b3-dfd7-4c1e-9453-a86f4a626df5\") " pod="kuadrant-system/authorino-5f5b44b88f-w7cpd" Jul 07 14:51:14.578083 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:14.578048 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/497257b3-dfd7-4c1e-9453-a86f4a626df5-tls-cert\") pod \"authorino-5f5b44b88f-w7cpd\" (UID: \"497257b3-dfd7-4c1e-9453-a86f4a626df5\") " pod="kuadrant-system/authorino-5f5b44b88f-w7cpd" Jul 07 14:51:14.578220 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:14.578094 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b98vk\" (UniqueName: \"kubernetes.io/projected/497257b3-dfd7-4c1e-9453-a86f4a626df5-kube-api-access-b98vk\") pod \"authorino-5f5b44b88f-w7cpd\" (UID: \"497257b3-dfd7-4c1e-9453-a86f4a626df5\") " pod="kuadrant-system/authorino-5f5b44b88f-w7cpd" Jul 07 14:51:14.580510 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:14.580481 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/497257b3-dfd7-4c1e-9453-a86f4a626df5-tls-cert\") pod \"authorino-5f5b44b88f-w7cpd\" (UID: \"497257b3-dfd7-4c1e-9453-a86f4a626df5\") " pod="kuadrant-system/authorino-5f5b44b88f-w7cpd" Jul 07 14:51:14.595173 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:14.595143 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b98vk\" (UniqueName: \"kubernetes.io/projected/497257b3-dfd7-4c1e-9453-a86f4a626df5-kube-api-access-b98vk\") pod \"authorino-5f5b44b88f-w7cpd\" (UID: \"497257b3-dfd7-4c1e-9453-a86f4a626df5\") " pod="kuadrant-system/authorino-5f5b44b88f-w7cpd" Jul 07 14:51:14.739090 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:14.739067 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5b44b88f-w7cpd" Jul 07 14:51:14.805196 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:14.805160 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:51:14.873482 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:14.873458 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f5b44b88f-w7cpd"] Jul 07 14:51:14.875531 ip-10-0-142-144 kubenswrapper[2583]: W0707 14:51:14.875501 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod497257b3_dfd7_4c1e_9453_a86f4a626df5.slice/crio-d43bf24b2f92f395dd3d1c7d0adf4903b673a4b600bf351456bb3f6e8b91f261 WatchSource:0}: Error finding container d43bf24b2f92f395dd3d1c7d0adf4903b673a4b600bf351456bb3f6e8b91f261: Status 404 returned error can't find the container with id d43bf24b2f92f395dd3d1c7d0adf4903b673a4b600bf351456bb3f6e8b91f261 Jul 07 14:51:14.876883 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:14.876867 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:51:14.913715 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:14.913692 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:51:15.679059 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:15.679021 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f5b44b88f-w7cpd" event={"ID":"497257b3-dfd7-4c1e-9453-a86f4a626df5","Type":"ContainerStarted","Data":"5caca662c48c4ee0e5542cd331c150f7badecdafa3237ea9b54308921d96835c"} Jul 07 14:51:15.679059 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:15.679064 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f5b44b88f-w7cpd" event={"ID":"497257b3-dfd7-4c1e-9453-a86f4a626df5","Type":"ContainerStarted","Data":"d43bf24b2f92f395dd3d1c7d0adf4903b673a4b600bf351456bb3f6e8b91f261"} Jul 07 14:51:15.708904 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:15.708849 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f5b44b88f-w7cpd" podStartSLOduration=1.082295662 podStartE2EDuration="1.708829954s" totalImagesPullingTime="626.534292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:51:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:51:15.706731719 +0000 UTC m=+1008.617925399" watchObservedRunningTime="2026-07-07 14:51:15.708829954 +0000 UTC m=+1008.620023634" Jul 07 14:51:15.768135 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:15.768056 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-657bf6847b-gxh8v"] Jul 07 14:51:15.768272 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:15.768235 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-657bf6847b-gxh8v" podUID="5ccb1371-d228-4ed6-bc88-a35ce5f6bc40" containerName="authorino" containerID="cri-o://47d2c1b48c312ea811ae3e4f22ca55ad97748566ae94b03bb1677d3793a58bbc" gracePeriod=30 Jul 07 14:51:16.022457 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:16.022395 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-657bf6847b-gxh8v" Jul 07 14:51:16.040490 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:16.040462 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:51:16.091282 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:16.091257 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5ccb1371-d228-4ed6-bc88-a35ce5f6bc40-tls-cert\") pod \"5ccb1371-d228-4ed6-bc88-a35ce5f6bc40\" (UID: \"5ccb1371-d228-4ed6-bc88-a35ce5f6bc40\") " Jul 07 14:51:16.091449 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:16.091335 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtgq9\" (UniqueName: \"kubernetes.io/projected/5ccb1371-d228-4ed6-bc88-a35ce5f6bc40-kube-api-access-rtgq9\") pod \"5ccb1371-d228-4ed6-bc88-a35ce5f6bc40\" (UID: \"5ccb1371-d228-4ed6-bc88-a35ce5f6bc40\") " Jul 07 14:51:16.093498 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:16.093468 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ccb1371-d228-4ed6-bc88-a35ce5f6bc40-kube-api-access-rtgq9" (OuterVolumeSpecName: "kube-api-access-rtgq9") pod "5ccb1371-d228-4ed6-bc88-a35ce5f6bc40" (UID: "5ccb1371-d228-4ed6-bc88-a35ce5f6bc40"). InnerVolumeSpecName "kube-api-access-rtgq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:51:16.102273 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:16.102246 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ccb1371-d228-4ed6-bc88-a35ce5f6bc40-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "5ccb1371-d228-4ed6-bc88-a35ce5f6bc40" (UID: "5ccb1371-d228-4ed6-bc88-a35ce5f6bc40"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:51:16.193041 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:16.193001 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtgq9\" (UniqueName: \"kubernetes.io/projected/5ccb1371-d228-4ed6-bc88-a35ce5f6bc40-kube-api-access-rtgq9\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:51:16.193041 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:16.193032 2583 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5ccb1371-d228-4ed6-bc88-a35ce5f6bc40-tls-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 14:51:16.217802 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:16.217767 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:51:16.685702 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:16.685668 2583 generic.go:356] "Generic (PLEG): container finished" podID="5ccb1371-d228-4ed6-bc88-a35ce5f6bc40" containerID="47d2c1b48c312ea811ae3e4f22ca55ad97748566ae94b03bb1677d3793a58bbc" exitCode=0 Jul 07 14:51:16.686174 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:16.685714 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-657bf6847b-gxh8v" Jul 07 14:51:16.686174 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:16.685750 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-657bf6847b-gxh8v" event={"ID":"5ccb1371-d228-4ed6-bc88-a35ce5f6bc40","Type":"ContainerDied","Data":"47d2c1b48c312ea811ae3e4f22ca55ad97748566ae94b03bb1677d3793a58bbc"} Jul 07 14:51:16.686174 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:16.685794 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-657bf6847b-gxh8v" event={"ID":"5ccb1371-d228-4ed6-bc88-a35ce5f6bc40","Type":"ContainerDied","Data":"ae8e79a58db79c7f145054876fad2ec5fe01c86f7c2b9691e01ccf6234bc4c88"} Jul 07 14:51:16.686174 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:16.685822 2583 scope.go:117] "RemoveContainer" containerID="47d2c1b48c312ea811ae3e4f22ca55ad97748566ae94b03bb1677d3793a58bbc" Jul 07 14:51:16.704009 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:16.703989 2583 scope.go:117] "RemoveContainer" containerID="47d2c1b48c312ea811ae3e4f22ca55ad97748566ae94b03bb1677d3793a58bbc" Jul 07 14:51:16.704269 ip-10-0-142-144 kubenswrapper[2583]: E0707 14:51:16.704250 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d2c1b48c312ea811ae3e4f22ca55ad97748566ae94b03bb1677d3793a58bbc\": container with ID starting with 47d2c1b48c312ea811ae3e4f22ca55ad97748566ae94b03bb1677d3793a58bbc not found: ID does not exist" containerID="47d2c1b48c312ea811ae3e4f22ca55ad97748566ae94b03bb1677d3793a58bbc" Jul 07 14:51:16.704337 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:16.704281 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d2c1b48c312ea811ae3e4f22ca55ad97748566ae94b03bb1677d3793a58bbc"} err="failed to get container status \"47d2c1b48c312ea811ae3e4f22ca55ad97748566ae94b03bb1677d3793a58bbc\": rpc error: code = NotFound desc = could not find container \"47d2c1b48c312ea811ae3e4f22ca55ad97748566ae94b03bb1677d3793a58bbc\": container with ID starting with 47d2c1b48c312ea811ae3e4f22ca55ad97748566ae94b03bb1677d3793a58bbc not found: ID does not exist" Jul 07 14:51:16.719270 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:16.719244 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-657bf6847b-gxh8v"] Jul 07 14:51:16.723003 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:16.722982 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-657bf6847b-gxh8v"] Jul 07 14:51:17.793026 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:17.792994 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ccb1371-d228-4ed6-bc88-a35ce5f6bc40" path="/var/lib/kubelet/pods/5ccb1371-d228-4ed6-bc88-a35ce5f6bc40/volumes" Jul 07 14:51:44.752045 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:44.752006 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:51:44.844763 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:44.844711 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:51:46.399006 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:46.398974 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:51:46.641093 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:51:46.641060 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:26.728838 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:52:26.728717 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:28.180631 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:52:28.180586 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:56.769105 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:52:56.769071 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:56.842735 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:52:56.842707 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:01.755607 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:53:01.755574 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:09.074233 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:53:09.074203 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:09.172692 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:53:09.172655 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:09.343792 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:53:09.343706 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:15.201472 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:53:15.201438 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:15.304607 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:53:15.304574 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:15.409218 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:53:15.409188 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:15.520686 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:53:15.520650 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:15.658030 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:53:15.657997 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:21.527408 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:53:21.527373 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:21.602276 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:53:21.602233 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:54:19.761756 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:54:19.761717 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:54:20.973860 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:54:20.973821 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:54:27.612113 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:54:27.612084 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 14:54:27.616988 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:54:27.616963 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 14:57:35.581181 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:57:35.581127 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:57:35.690105 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:57:35.690073 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:27.646734 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:59:27.646709 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 14:59:27.652676 ip-10-0-142-144 kubenswrapper[2583]: I0707 14:59:27.652655 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 15:00:00.155990 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:00.155960 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29723940-rd9nj"] Jul 07 15:00:00.156480 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:00.156259 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ccb1371-d228-4ed6-bc88-a35ce5f6bc40" containerName="authorino" Jul 07 15:00:00.156480 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:00.156270 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ccb1371-d228-4ed6-bc88-a35ce5f6bc40" containerName="authorino" Jul 07 15:00:00.156480 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:00.156316 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ccb1371-d228-4ed6-bc88-a35ce5f6bc40" containerName="authorino" Jul 07 15:00:00.158759 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:00.158743 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723940-rd9nj" Jul 07 15:00:00.161449 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:00.161429 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-gjhm4\"" Jul 07 15:00:00.167789 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:00.167766 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29723940-rd9nj"] Jul 07 15:00:00.216450 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:00.216424 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgh5b\" (UniqueName: \"kubernetes.io/projected/835568d6-74c8-48f0-8e21-e4550d1a45d1-kube-api-access-bgh5b\") pod \"maas-api-key-cleanup-29723940-rd9nj\" (UID: \"835568d6-74c8-48f0-8e21-e4550d1a45d1\") " pod="opendatahub/maas-api-key-cleanup-29723940-rd9nj" Jul 07 15:00:00.317295 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:00.317270 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgh5b\" (UniqueName: \"kubernetes.io/projected/835568d6-74c8-48f0-8e21-e4550d1a45d1-kube-api-access-bgh5b\") pod \"maas-api-key-cleanup-29723940-rd9nj\" (UID: \"835568d6-74c8-48f0-8e21-e4550d1a45d1\") " pod="opendatahub/maas-api-key-cleanup-29723940-rd9nj" Jul 07 15:00:00.328634 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:00.328613 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgh5b\" (UniqueName: \"kubernetes.io/projected/835568d6-74c8-48f0-8e21-e4550d1a45d1-kube-api-access-bgh5b\") pod \"maas-api-key-cleanup-29723940-rd9nj\" (UID: \"835568d6-74c8-48f0-8e21-e4550d1a45d1\") " pod="opendatahub/maas-api-key-cleanup-29723940-rd9nj" Jul 07 15:00:00.469246 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:00.469223 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723940-rd9nj" Jul 07 15:00:00.597203 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:00.597176 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29723940-rd9nj"] Jul 07 15:00:00.598504 ip-10-0-142-144 kubenswrapper[2583]: W0707 15:00:00.598481 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835568d6_74c8_48f0_8e21_e4550d1a45d1.slice/crio-2a8cb736dbec264585d1400fc1e177811cae17b972eb00b4bb55dc868ae997c4 WatchSource:0}: Error finding container 2a8cb736dbec264585d1400fc1e177811cae17b972eb00b4bb55dc868ae997c4: Status 404 returned error can't find the container with id 2a8cb736dbec264585d1400fc1e177811cae17b972eb00b4bb55dc868ae997c4 Jul 07 15:00:00.600334 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:00.600314 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 15:00:01.146201 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:01.146169 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723940-rd9nj" event={"ID":"835568d6-74c8-48f0-8e21-e4550d1a45d1","Type":"ContainerStarted","Data":"2a8cb736dbec264585d1400fc1e177811cae17b972eb00b4bb55dc868ae997c4"} Jul 07 15:00:07.177166 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:07.177101 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723940-rd9nj" event={"ID":"835568d6-74c8-48f0-8e21-e4550d1a45d1","Type":"ContainerStarted","Data":"ad4b86cce5305badfc29a6aa846106cb1b90caa461f700365105c6c5ecb16115"} Jul 07 15:00:07.195032 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:07.194981 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29723940-rd9nj" podStartSLOduration=0.978758657 podStartE2EDuration="7.19496902s" totalImagesPullingTime="6.216210363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:00:07.194328638 +0000 UTC m=+1540.105522316" watchObservedRunningTime="2026-07-07 15:00:07.19496902 +0000 UTC m=+1540.106162697" Jul 07 15:00:08.184263 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:08.184224 2583 generic.go:356] "Generic (PLEG): container finished" podID="835568d6-74c8-48f0-8e21-e4550d1a45d1" containerID="ad4b86cce5305badfc29a6aa846106cb1b90caa461f700365105c6c5ecb16115" exitCode=0 Jul 07 15:00:08.184664 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:08.184327 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723940-rd9nj" event={"ID":"835568d6-74c8-48f0-8e21-e4550d1a45d1","Type":"ContainerDied","Data":"ad4b86cce5305badfc29a6aa846106cb1b90caa461f700365105c6c5ecb16115"} Jul 07 15:00:09.324786 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:09.324759 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723940-rd9nj" Jul 07 15:00:09.376527 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:09.376494 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgh5b\" (UniqueName: \"kubernetes.io/projected/835568d6-74c8-48f0-8e21-e4550d1a45d1-kube-api-access-bgh5b\") pod \"835568d6-74c8-48f0-8e21-e4550d1a45d1\" (UID: \"835568d6-74c8-48f0-8e21-e4550d1a45d1\") " Jul 07 15:00:09.378740 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:09.378710 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835568d6-74c8-48f0-8e21-e4550d1a45d1-kube-api-access-bgh5b" (OuterVolumeSpecName: "kube-api-access-bgh5b") pod "835568d6-74c8-48f0-8e21-e4550d1a45d1" (UID: "835568d6-74c8-48f0-8e21-e4550d1a45d1"). InnerVolumeSpecName "kube-api-access-bgh5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:00:09.477129 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:09.477099 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgh5b\" (UniqueName: \"kubernetes.io/projected/835568d6-74c8-48f0-8e21-e4550d1a45d1-kube-api-access-bgh5b\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:00:10.195591 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:10.195558 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723940-rd9nj" event={"ID":"835568d6-74c8-48f0-8e21-e4550d1a45d1","Type":"ContainerDied","Data":"2a8cb736dbec264585d1400fc1e177811cae17b972eb00b4bb55dc868ae997c4"} Jul 07 15:00:10.195728 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:10.195610 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a8cb736dbec264585d1400fc1e177811cae17b972eb00b4bb55dc868ae997c4" Jul 07 15:00:10.195728 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:10.195574 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723940-rd9nj" Jul 07 15:00:42.793158 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:42.793122 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:00:54.512131 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:00:54.512097 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:01:45.015254 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:01:45.015167 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:18.441713 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:18.441679 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:18.609968 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:18.609914 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:18.867629 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:18.867592 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:24.907532 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:24.907499 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:25.015467 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:25.015434 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:25.099117 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:25.099090 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:26.486110 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:26.486078 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:26.685141 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:26.685113 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:26.946758 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:26.946665 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:27.046650 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:27.046602 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:27.677802 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:27.677702 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 15:04:27.689425 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:27.689404 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 15:04:28.817730 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:28.817701 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:31.406840 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:31.406805 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:31.660115 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:31.660028 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:31.750099 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:31.750063 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:32.277614 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:32.277577 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:33.011603 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:33.011553 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:37.968845 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:37.968813 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:38.072070 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:38.072038 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:38.169936 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:38.169905 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:39.281839 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:39.281805 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:39.385057 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:39.385025 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:39.723760 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:39.723725 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:39.820421 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:39.820389 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:39.920406 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:39.920375 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:40.121216 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:40.121140 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:41.290588 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:41.290554 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:44.361245 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:44.361184 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:44.818594 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:44.818563 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:44.879209 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:44.879181 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:50.501007 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:50.500977 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:50.614556 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:50.614522 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:52.385300 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:52.385268 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:56.716082 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:56.716048 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:56.816941 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:56.816905 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:56.947972 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:56.947944 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:57.049947 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:57.049916 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:58.055601 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:58.055568 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:58.168464 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:58.168432 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:58.255479 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:58.255443 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:58.746715 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:58.746681 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:59.000227 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:59.000139 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:59.128767 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:04:59.128738 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:00.787552 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:00.787519 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:02.526623 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:02.526587 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:02.722979 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:02.722947 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:02.822677 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:02.822561 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:02.957041 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:02.957008 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:04.259099 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:04.259060 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:08.984898 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:08.984864 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:09.049792 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:09.049759 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:09.140019 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:09.139983 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:09.487831 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:09.487802 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:09.587642 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:09.587609 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:10.775515 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:10.775475 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:16.776101 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:16.776060 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:16.879417 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:16.879384 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:16.980932 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:16.980898 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:17.080111 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:17.080033 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:17.569263 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:17.569228 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:43.533940 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:43.533906 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:49.400286 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:49.400204 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:51.172868 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:05:51.172827 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:05.559873 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:05.559837 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:11.346516 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:11.346478 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:11.413349 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:11.413303 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:11.507674 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:11.507639 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:11.603442 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:11.603338 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:22.117214 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:22.117179 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:22.967888 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:22.967857 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:29.030244 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:29.030210 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:29.136251 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:29.136219 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:29.230799 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:29.230763 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:29.331143 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:29.331059 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:30.167036 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:30.166999 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:30.258253 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:30.258225 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:30.362772 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:30.362747 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:32.005676 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:32.005644 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:37.912807 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:37.912774 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:37.977475 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:37.977438 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:38.633988 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:38.633953 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:44.344812 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:44.344777 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:44.409562 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:44.409508 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:44.649788 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:44.649702 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:44.958152 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:44.958072 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:45.115643 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:45.115594 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:48.273268 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:48.273234 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:48.380274 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:48.380229 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:48.533196 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:48.533129 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:48.855066 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:48.854977 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:49.309986 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:49.309953 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:49.486421 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:49.486387 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:52.790575 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:52.790541 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:53.474558 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:53.474527 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:53.574548 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:53.574519 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:53.934145 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:53.934062 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:59.809145 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:59.809086 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:59.985404 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:06:59.985348 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:00.209066 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:00.208994 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:00.569510 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:00.569482 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:00.643089 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:00.643059 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:06.806985 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:06.806948 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:06.905882 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:06.905850 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:07.350848 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:07.350812 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:07.459122 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:07.459078 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:07.560682 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:07.560650 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:07.664156 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:07.664084 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:13.410584 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:13.410556 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:13.512866 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:13.512830 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:13.610566 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:13.610534 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:13.767779 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:13.767744 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:19.580401 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:19.580308 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:19.765032 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:19.765000 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:26.514537 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:26.514500 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:26.853767 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:26.853693 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:27.347070 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:27.347035 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:29.436042 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:29.436002 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:29.528680 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:29.528653 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:29.667439 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:29.667407 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:29.939592 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:29.939555 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:30.207189 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:30.207114 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:30.317552 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:30.317509 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:33.948975 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:33.948944 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:43.959744 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:43.959692 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp"] Jul 07 15:07:43.960138 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:43.959987 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="835568d6-74c8-48f0-8e21-e4550d1a45d1" containerName="cleanup" Jul 07 15:07:43.960138 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:43.959998 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="835568d6-74c8-48f0-8e21-e4550d1a45d1" containerName="cleanup" Jul 07 15:07:43.960138 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:43.960063 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="835568d6-74c8-48f0-8e21-e4550d1a45d1" containerName="cleanup" Jul 07 15:07:43.961991 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:43.961975 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:43.965090 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:43.965061 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-09af33f3\"/\"kube-root-ca.crt\"" Jul 07 15:07:43.965213 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:43.965146 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-09af33f3\"/\"openshift-service-ca.crt\"" Jul 07 15:07:43.965213 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:43.965170 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-09af33f3\"/\"e2e-lifecycle-model-09af33f3-kserve-self-signed-certs\"" Jul 07 15:07:43.965342 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:43.965327 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-09af33f3\"/\"default-dockercfg-blsst\"" Jul 07 15:07:43.973082 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:43.973062 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp"] Jul 07 15:07:43.984186 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:43.984164 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:44.014004 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.013978 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-dshm\") pod \"e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:44.014106 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.014021 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dabcdbb1-58f1-4440-83e4-d347c66b46e9-tls-certs\") pod \"e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:44.014106 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.014048 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj7ph\" (UniqueName: \"kubernetes.io/projected/dabcdbb1-58f1-4440-83e4-d347c66b46e9-kube-api-access-bj7ph\") pod \"e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:44.014182 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.014105 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-home\") pod \"e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:44.014182 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.014141 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-tmp-dir\") pod \"e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:44.014182 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.014174 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-model-cache\") pod \"e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:44.115022 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.114994 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-tmp-dir\") pod \"e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:44.115154 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.115033 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-model-cache\") pod \"e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:44.115154 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.115070 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-dshm\") pod \"e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:44.115154 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.115097 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dabcdbb1-58f1-4440-83e4-d347c66b46e9-tls-certs\") pod \"e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:44.115154 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.115113 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj7ph\" (UniqueName: \"kubernetes.io/projected/dabcdbb1-58f1-4440-83e4-d347c66b46e9-kube-api-access-bj7ph\") pod \"e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:44.115154 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.115132 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-home\") pod \"e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:44.115513 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.115478 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-model-cache\") pod \"e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:44.115513 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.115503 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-tmp-dir\") pod \"e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:44.115664 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.115548 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-home\") pod \"e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:44.117306 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.117285 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-dshm\") pod \"e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:44.117731 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.117713 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dabcdbb1-58f1-4440-83e4-d347c66b46e9-tls-certs\") pod \"e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:44.124202 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.124180 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj7ph\" (UniqueName: \"kubernetes.io/projected/dabcdbb1-58f1-4440-83e4-d347c66b46e9-kube-api-access-bj7ph\") pod \"e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:44.270861 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.270821 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:44.402310 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:44.402283 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp"] Jul 07 15:07:44.404076 ip-10-0-142-144 kubenswrapper[2583]: W0707 15:07:44.404053 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddabcdbb1_58f1_4440_83e4_d347c66b46e9.slice/crio-86155708dcfcb3a42ea5cc411714a1eecdc11164dfddce70205ca8dad5203750 WatchSource:0}: Error finding container 86155708dcfcb3a42ea5cc411714a1eecdc11164dfddce70205ca8dad5203750: Status 404 returned error can't find the container with id 86155708dcfcb3a42ea5cc411714a1eecdc11164dfddce70205ca8dad5203750 Jul 07 15:07:45.318602 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:45.318567 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" event={"ID":"dabcdbb1-58f1-4440-83e4-d347c66b46e9","Type":"ContainerStarted","Data":"1cce6fd1666bb7c49f71b232cf4ebcf1be7ff8980ec586a250215ac6377866de"} Jul 07 15:07:45.318602 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:45.318607 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" event={"ID":"dabcdbb1-58f1-4440-83e4-d347c66b46e9","Type":"ContainerStarted","Data":"86155708dcfcb3a42ea5cc411714a1eecdc11164dfddce70205ca8dad5203750"} Jul 07 15:07:45.337904 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:45.337865 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" podStartSLOduration=2.337851335 podStartE2EDuration="2.337851335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:07:45.335970968 +0000 UTC m=+1998.247164645" watchObservedRunningTime="2026-07-07 15:07:45.337851335 +0000 UTC m=+1998.249045012" Jul 07 15:07:49.976977 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:49.976943 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:50.343038 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:50.342955 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:50.440331 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:50.440293 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:51.054230 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:51.054198 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:51.146697 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:51.146668 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:07:54.271825 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:54.271793 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:54.272192 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:54.271873 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:54.284057 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:54.284034 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:54.371005 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:54.370983 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:07:55.901634 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:07:55.901598 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:08:01.333816 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:01.333782 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:08:01.984953 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:01.984917 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:08:07.516857 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:07.516820 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:08:12.490307 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:12.490271 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp"] Jul 07 15:08:12.587194 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:12.587166 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:08:27.510221 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.510178 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" podUID="dabcdbb1-58f1-4440-83e4-d347c66b46e9" containerName="main" containerID="cri-o://1cce6fd1666bb7c49f71b232cf4ebcf1be7ff8980ec586a250215ac6377866de" gracePeriod=45 Jul 07 15:08:27.750459 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.750433 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:08:27.851673 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.851601 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bj7ph\" (UniqueName: \"kubernetes.io/projected/dabcdbb1-58f1-4440-83e4-d347c66b46e9-kube-api-access-bj7ph\") pod \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " Jul 07 15:08:27.851819 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.851684 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dabcdbb1-58f1-4440-83e4-d347c66b46e9-tls-certs\") pod \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " Jul 07 15:08:27.851819 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.851720 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-model-cache\") pod \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " Jul 07 15:08:27.851819 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.851750 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-tmp-dir\") pod \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " Jul 07 15:08:27.851819 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.851775 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-home\") pod \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " Jul 07 15:08:27.852029 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.851844 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-dshm\") pod \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\" (UID: \"dabcdbb1-58f1-4440-83e4-d347c66b46e9\") " Jul 07 15:08:27.852029 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.851989 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "dabcdbb1-58f1-4440-83e4-d347c66b46e9" (UID: "dabcdbb1-58f1-4440-83e4-d347c66b46e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:08:27.852029 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.852014 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-home" (OuterVolumeSpecName: "home") pod "dabcdbb1-58f1-4440-83e4-d347c66b46e9" (UID: "dabcdbb1-58f1-4440-83e4-d347c66b46e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:08:27.852238 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.852037 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dabcdbb1-58f1-4440-83e4-d347c66b46e9" (UID: "dabcdbb1-58f1-4440-83e4-d347c66b46e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:08:27.852238 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.852104 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-model-cache\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:08:27.852238 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.852118 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-tmp-dir\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:08:27.852238 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.852129 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-home\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:08:27.853729 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.853705 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-dshm" (OuterVolumeSpecName: "dshm") pod "dabcdbb1-58f1-4440-83e4-d347c66b46e9" (UID: "dabcdbb1-58f1-4440-83e4-d347c66b46e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:08:27.853818 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.853789 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dabcdbb1-58f1-4440-83e4-d347c66b46e9-kube-api-access-bj7ph" (OuterVolumeSpecName: "kube-api-access-bj7ph") pod "dabcdbb1-58f1-4440-83e4-d347c66b46e9" (UID: "dabcdbb1-58f1-4440-83e4-d347c66b46e9"). InnerVolumeSpecName "kube-api-access-bj7ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:08:27.853876 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.853863 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dabcdbb1-58f1-4440-83e4-d347c66b46e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dabcdbb1-58f1-4440-83e4-d347c66b46e9" (UID: "dabcdbb1-58f1-4440-83e4-d347c66b46e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:08:27.952619 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.952595 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dabcdbb1-58f1-4440-83e4-d347c66b46e9-dshm\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:08:27.952619 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.952617 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj7ph\" (UniqueName: \"kubernetes.io/projected/dabcdbb1-58f1-4440-83e4-d347c66b46e9-kube-api-access-bj7ph\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:08:27.952758 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:27.952627 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dabcdbb1-58f1-4440-83e4-d347c66b46e9-tls-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:08:28.520459 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:28.520427 2583 generic.go:356] "Generic (PLEG): container finished" podID="dabcdbb1-58f1-4440-83e4-d347c66b46e9" containerID="1cce6fd1666bb7c49f71b232cf4ebcf1be7ff8980ec586a250215ac6377866de" exitCode=0 Jul 07 15:08:28.520882 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:28.520470 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" event={"ID":"dabcdbb1-58f1-4440-83e4-d347c66b46e9","Type":"ContainerDied","Data":"1cce6fd1666bb7c49f71b232cf4ebcf1be7ff8980ec586a250215ac6377866de"} Jul 07 15:08:28.520882 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:28.520498 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" event={"ID":"dabcdbb1-58f1-4440-83e4-d347c66b46e9","Type":"ContainerDied","Data":"86155708dcfcb3a42ea5cc411714a1eecdc11164dfddce70205ca8dad5203750"} Jul 07 15:08:28.520882 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:28.520515 2583 scope.go:117] "RemoveContainer" containerID="1cce6fd1666bb7c49f71b232cf4ebcf1be7ff8980ec586a250215ac6377866de" Jul 07 15:08:28.520882 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:28.520472 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp" Jul 07 15:08:28.529435 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:28.529422 2583 scope.go:117] "RemoveContainer" containerID="1cce6fd1666bb7c49f71b232cf4ebcf1be7ff8980ec586a250215ac6377866de" Jul 07 15:08:28.529683 ip-10-0-142-144 kubenswrapper[2583]: E0707 15:08:28.529663 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cce6fd1666bb7c49f71b232cf4ebcf1be7ff8980ec586a250215ac6377866de\": container with ID starting with 1cce6fd1666bb7c49f71b232cf4ebcf1be7ff8980ec586a250215ac6377866de not found: ID does not exist" containerID="1cce6fd1666bb7c49f71b232cf4ebcf1be7ff8980ec586a250215ac6377866de" Jul 07 15:08:28.529757 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:28.529690 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cce6fd1666bb7c49f71b232cf4ebcf1be7ff8980ec586a250215ac6377866de"} err="failed to get container status \"1cce6fd1666bb7c49f71b232cf4ebcf1be7ff8980ec586a250215ac6377866de\": rpc error: code = NotFound desc = could not find container \"1cce6fd1666bb7c49f71b232cf4ebcf1be7ff8980ec586a250215ac6377866de\": container with ID starting with 1cce6fd1666bb7c49f71b232cf4ebcf1be7ff8980ec586a250215ac6377866de not found: ID does not exist" Jul 07 15:08:28.549728 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:28.549699 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp"] Jul 07 15:08:28.558038 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:28.556009 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-09af33f3/e2e-lifecycle-model-09af33f3-kserve-676c99ccc5-pz4kp"] Jul 07 15:08:29.791826 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:29.791793 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dabcdbb1-58f1-4440-83e4-d347c66b46e9" path="/var/lib/kubelet/pods/dabcdbb1-58f1-4440-83e4-d347c66b46e9/volumes" Jul 07 15:08:50.080518 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:50.080439 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:08:50.882932 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:50.882901 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:08:58.122813 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:08:58.122781 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:03.947617 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:03.947573 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:04.084595 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:04.084560 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:04.296645 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:04.296607 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:04.590470 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:04.590387 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:05.789001 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:05.788965 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:05.909640 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:05.909611 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:06.784376 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:06.784340 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:06.878097 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:06.878064 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:06.984346 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:06.984316 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:08.487190 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:08.487157 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:15.095554 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:15.095520 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:15.162580 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:15.162550 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:15.820641 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:15.820604 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:21.859326 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:21.859293 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:22.314112 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:22.314079 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:22.471815 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:22.471779 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:27.713712 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:27.713597 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 15:09:27.720705 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:27.720685 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 15:09:32.622641 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:32.622605 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:32.989347 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:32.989307 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:53.992378 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:53.992323 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:56.072731 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:56.072699 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:09:56.156476 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:09:56.156430 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:00.176563 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:00.176527 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:00.279660 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:00.279619 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:00.536689 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:00.536659 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:00.866879 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:00.866804 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:00.957377 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:00.957332 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:04.218639 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:04.218608 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:04.389702 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:04.389666 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:04.489299 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:04.489213 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:04.789572 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:04.789484 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:04.939261 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:04.939217 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:05.066181 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:05.066093 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:08.735692 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:08.735654 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:09.080807 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.080718 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:09.378312 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.378237 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9"] Jul 07 15:10:09.378717 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.378702 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dabcdbb1-58f1-4440-83e4-d347c66b46e9" containerName="main" Jul 07 15:10:09.378764 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.378721 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dabcdbb1-58f1-4440-83e4-d347c66b46e9" containerName="main" Jul 07 15:10:09.378827 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.378817 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dabcdbb1-58f1-4440-83e4-d347c66b46e9" containerName="main" Jul 07 15:10:09.382032 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.382013 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.384488 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.384465 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c95db0-openshift-default-dockercfg-cdm26\"" Jul 07 15:10:09.395540 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.395521 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9"] Jul 07 15:10:09.424005 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.423984 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:09.449287 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.449265 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-envoy\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.449394 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.449295 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-workload-certs\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.449394 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.449325 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fa5688-ba15-4be8-97c7-b786af8587ce-istiod-ca-cert\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.449470 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.449444 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-token\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.449470 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.449461 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-credential-socket\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.449539 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.449477 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-data\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.449539 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.449504 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-podinfo\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.449539 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.449520 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtqcc\" (UniqueName: \"kubernetes.io/projected/35fa5688-ba15-4be8-97c7-b786af8587ce-kube-api-access-xtqcc\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.449637 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.449538 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-workload-socket\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.550240 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.550206 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-credential-socket\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.550240 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.550238 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-data\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.550517 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.550257 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-podinfo\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.550517 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.550274 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtqcc\" (UniqueName: \"kubernetes.io/projected/35fa5688-ba15-4be8-97c7-b786af8587ce-kube-api-access-xtqcc\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.550517 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.550297 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-workload-socket\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.550517 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.550350 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-envoy\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.550517 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.550404 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-workload-certs\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.550517 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.550448 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fa5688-ba15-4be8-97c7-b786af8587ce-istiod-ca-cert\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.550838 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.550541 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-token\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.550838 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.550676 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-data\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.550838 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.550717 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-credential-socket\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.550838 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.550762 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-workload-socket\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.551039 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.550937 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-workload-certs\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.551251 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.551234 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fa5688-ba15-4be8-97c7-b786af8587ce-istiod-ca-cert\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.553149 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.553126 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-envoy\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.553273 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.553255 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-podinfo\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.559582 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.559557 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-token\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.559866 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.559848 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtqcc\" (UniqueName: \"kubernetes.io/projected/35fa5688-ba15-4be8-97c7-b786af8587ce-kube-api-access-xtqcc\") pod \"e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.644086 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.644015 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:09.692065 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.692033 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:09.826094 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.826066 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9"] Jul 07 15:10:09.828182 ip-10-0-142-144 kubenswrapper[2583]: W0707 15:10:09.828157 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35fa5688_ba15_4be8_97c7_b786af8587ce.slice/crio-5997b7f419984c145a25968ab06a987f3de215d0d2cf27d9d80cb252bceee2d9 WatchSource:0}: Error finding container 5997b7f419984c145a25968ab06a987f3de215d0d2cf27d9d80cb252bceee2d9: Status 404 returned error can't find the container with id 5997b7f419984c145a25968ab06a987f3de215d0d2cf27d9d80cb252bceee2d9 Jul 07 15:10:09.830169 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.830140 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:10:09.830239 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.830207 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:10:09.830239 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.830236 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:10:09.992751 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.992716 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" event={"ID":"35fa5688-ba15-4be8-97c7-b786af8587ce","Type":"ContainerStarted","Data":"c3d88811a07a308164ed5e7be40962bf29b45c359ef30db50cc8f19ee9cb89b2"} Jul 07 15:10:09.992924 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:09.992774 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" event={"ID":"35fa5688-ba15-4be8-97c7-b786af8587ce","Type":"ContainerStarted","Data":"5997b7f419984c145a25968ab06a987f3de215d0d2cf27d9d80cb252bceee2d9"} Jul 07 15:10:10.019090 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:10.019036 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" podStartSLOduration=1.019019354 podStartE2EDuration="1.019019354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:10:10.017805823 +0000 UTC m=+2142.928999506" watchObservedRunningTime="2026-07-07 15:10:10.019019354 +0000 UTC m=+2142.930213057" Jul 07 15:10:10.692685 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:10.692643 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:10.697653 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:10.697629 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:10.998157 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:10.998129 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:10.999034 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:10.999014 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:11.985776 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:11.985738 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:12.083684 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:12.083626 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:12.254556 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:12.254437 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:12.544425 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:12.544314 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:12.806060 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:12.805971 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:12.905311 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:12.905282 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:16.504091 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:16.504060 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:23.260540 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.260502 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc"] Jul 07 15:10:23.265522 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.265499 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.269392 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.269195 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-45adb8\"/\"default-dockercfg-rh4s6\"" Jul 07 15:10:23.269392 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.269209 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-45adb8\"/\"openshift-service-ca.crt\"" Jul 07 15:10:23.269392 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.269242 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-45adb8\"/\"kube-root-ca.crt\"" Jul 07 15:10:23.269651 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.269515 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-45adb8\"/\"test-model-45adb8-kserve-self-signed-certs\"" Jul 07 15:10:23.276483 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.276462 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc"] Jul 07 15:10:23.297767 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.297740 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:23.381376 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.381324 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6xg7\" (UniqueName: \"kubernetes.io/projected/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-kube-api-access-v6xg7\") pod \"test-model-45adb8-kserve-5d5fbf788f-7gtpc\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.381556 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.381487 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-model-cache\") pod \"test-model-45adb8-kserve-5d5fbf788f-7gtpc\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.381617 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.381569 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-dshm\") pod \"test-model-45adb8-kserve-5d5fbf788f-7gtpc\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.381617 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.381603 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-tmp-dir\") pod \"test-model-45adb8-kserve-5d5fbf788f-7gtpc\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.381701 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.381636 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-home\") pod \"test-model-45adb8-kserve-5d5fbf788f-7gtpc\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.381701 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.381668 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-tls-certs\") pod \"test-model-45adb8-kserve-5d5fbf788f-7gtpc\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.482689 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.482650 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6xg7\" (UniqueName: \"kubernetes.io/projected/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-kube-api-access-v6xg7\") pod \"test-model-45adb8-kserve-5d5fbf788f-7gtpc\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.482871 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.482748 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-model-cache\") pod \"test-model-45adb8-kserve-5d5fbf788f-7gtpc\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.482871 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.482793 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-dshm\") pod \"test-model-45adb8-kserve-5d5fbf788f-7gtpc\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.482871 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.482823 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-tmp-dir\") pod \"test-model-45adb8-kserve-5d5fbf788f-7gtpc\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.482871 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.482847 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-home\") pod \"test-model-45adb8-kserve-5d5fbf788f-7gtpc\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.482871 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.482868 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-tls-certs\") pod \"test-model-45adb8-kserve-5d5fbf788f-7gtpc\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.483235 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.483208 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-model-cache\") pod \"test-model-45adb8-kserve-5d5fbf788f-7gtpc\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.483317 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.483294 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-home\") pod \"test-model-45adb8-kserve-5d5fbf788f-7gtpc\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.483391 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.483341 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-tmp-dir\") pod \"test-model-45adb8-kserve-5d5fbf788f-7gtpc\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.485158 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.485132 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-dshm\") pod \"test-model-45adb8-kserve-5d5fbf788f-7gtpc\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.485432 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.485415 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-tls-certs\") pod \"test-model-45adb8-kserve-5d5fbf788f-7gtpc\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.491702 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.491676 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6xg7\" (UniqueName: \"kubernetes.io/projected/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-kube-api-access-v6xg7\") pod \"test-model-45adb8-kserve-5d5fbf788f-7gtpc\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.576973 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.576881 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:23.713842 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:23.713815 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc"] Jul 07 15:10:23.715891 ip-10-0-142-144 kubenswrapper[2583]: W0707 15:10:23.715862 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcaca07b_cb01_48fd_8b34_c8a9099ebce8.slice/crio-63d3b01dfcaf4320e1bb15f2f0b036b63d7f4a27cc5ceb55e6d5cdf60fb18d7b WatchSource:0}: Error finding container 63d3b01dfcaf4320e1bb15f2f0b036b63d7f4a27cc5ceb55e6d5cdf60fb18d7b: Status 404 returned error can't find the container with id 63d3b01dfcaf4320e1bb15f2f0b036b63d7f4a27cc5ceb55e6d5cdf60fb18d7b Jul 07 15:10:24.077880 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:24.077840 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" event={"ID":"fcaca07b-cb01-48fd-8b34-c8a9099ebce8","Type":"ContainerStarted","Data":"1700c3552c4e608c212529cba7f02b303f3ef523342d5111a834a67d843ca7a3"} Jul 07 15:10:24.077880 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:24.077886 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" event={"ID":"fcaca07b-cb01-48fd-8b34-c8a9099ebce8","Type":"ContainerStarted","Data":"63d3b01dfcaf4320e1bb15f2f0b036b63d7f4a27cc5ceb55e6d5cdf60fb18d7b"} Jul 07 15:10:24.095430 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:24.095388 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" podStartSLOduration=1.095374715 podStartE2EDuration="1.095374715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:10:24.093865421 +0000 UTC m=+2157.005059100" watchObservedRunningTime="2026-07-07 15:10:24.095374715 +0000 UTC m=+2157.006568384" Jul 07 15:10:25.708548 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:25.708449 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:25.804895 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:25.804861 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:25.999945 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:25.999917 2583 patch_prober.go:28] interesting pod/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 07 15:10:26.000107 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:25.999996 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" podUID="35fa5688-ba15-4be8-97c7-b786af8587ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 07 15:10:26.089438 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:26.089407 2583 generic.go:356] "Generic (PLEG): container finished" podID="35fa5688-ba15-4be8-97c7-b786af8587ce" containerID="c3d88811a07a308164ed5e7be40962bf29b45c359ef30db50cc8f19ee9cb89b2" exitCode=0 Jul 07 15:10:26.089602 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:26.089444 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" event={"ID":"35fa5688-ba15-4be8-97c7-b786af8587ce","Type":"ContainerDied","Data":"c3d88811a07a308164ed5e7be40962bf29b45c359ef30db50cc8f19ee9cb89b2"} Jul 07 15:10:26.089815 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:26.089801 2583 scope.go:117] "RemoveContainer" containerID="c3d88811a07a308164ed5e7be40962bf29b45c359ef30db50cc8f19ee9cb89b2" Jul 07 15:10:26.090840 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:26.090810 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:10:26.090899 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:26.090884 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:10:26.090936 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:26.090921 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:10:27.096516 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:27.096481 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" event={"ID":"35fa5688-ba15-4be8-97c7-b786af8587ce","Type":"ContainerStarted","Data":"02be96c4618a08f414fc866fa9ab5c0e06a4520fff21943d92c95fdeb560eb22"} Jul 07 15:10:27.693036 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:27.693000 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:27.693219 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:27.693189 2583 patch_prober.go:28] interesting pod/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 07 15:10:27.693268 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:27.693234 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" podUID="35fa5688-ba15-4be8-97c7-b786af8587ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 07 15:10:28.692474 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:28.692438 2583 patch_prober.go:28] interesting pod/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 07 15:10:28.692877 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:28.692494 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" podUID="35fa5688-ba15-4be8-97c7-b786af8587ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 07 15:10:29.693027 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:29.692997 2583 patch_prober.go:28] interesting pod/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 07 15:10:29.693421 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:29.693051 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" podUID="35fa5688-ba15-4be8-97c7-b786af8587ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 07 15:10:30.693456 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:30.693418 2583 patch_prober.go:28] interesting pod/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 07 15:10:30.693844 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:30.693478 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" podUID="35fa5688-ba15-4be8-97c7-b786af8587ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 07 15:10:31.696057 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:31.696027 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:31.696473 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:31.696284 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:31.696986 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:31.696968 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:10:33.577107 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:33.577077 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:33.577524 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:33.577122 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:33.589564 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:33.589539 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:34.142535 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:34.142504 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:10:34.713487 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:34.713453 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:34.819636 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:34.819606 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:49.394213 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.394177 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k"] Jul 07 15:10:49.396636 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.396618 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.399203 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.399181 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c95db0\"/\"openshift-service-ca.crt\"" Jul 07 15:10:49.399403 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.399385 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c95db0\"/\"test-model-c95db0-kserve-self-signed-certs\"" Jul 07 15:10:49.400298 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.400280 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c95db0\"/\"default-dockercfg-5mj5p\"" Jul 07 15:10:49.400630 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.400322 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c95db0\"/\"kube-root-ca.crt\"" Jul 07 15:10:49.409622 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.409601 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k"] Jul 07 15:10:49.468547 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.468518 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:10:49.509006 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.508984 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-home\") pod \"test-model-c95db0-kserve-b6dbcd78c-pr64k\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.509155 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.509018 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-dshm\") pod \"test-model-c95db0-kserve-b6dbcd78c-pr64k\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.509155 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.509036 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-tls-certs\") pod \"test-model-c95db0-kserve-b6dbcd78c-pr64k\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.509155 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.509057 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-model-cache\") pod \"test-model-c95db0-kserve-b6dbcd78c-pr64k\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.509155 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.509104 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-tmp-dir\") pod \"test-model-c95db0-kserve-b6dbcd78c-pr64k\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.509155 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.509147 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vqsr\" (UniqueName: \"kubernetes.io/projected/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-kube-api-access-9vqsr\") pod \"test-model-c95db0-kserve-b6dbcd78c-pr64k\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.609581 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.609548 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-home\") pod \"test-model-c95db0-kserve-b6dbcd78c-pr64k\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.609728 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.609598 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-dshm\") pod \"test-model-c95db0-kserve-b6dbcd78c-pr64k\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.609728 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.609620 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-tls-certs\") pod \"test-model-c95db0-kserve-b6dbcd78c-pr64k\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.609728 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.609649 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-model-cache\") pod \"test-model-c95db0-kserve-b6dbcd78c-pr64k\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.609728 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.609677 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-tmp-dir\") pod \"test-model-c95db0-kserve-b6dbcd78c-pr64k\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.609728 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.609715 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vqsr\" (UniqueName: \"kubernetes.io/projected/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-kube-api-access-9vqsr\") pod \"test-model-c95db0-kserve-b6dbcd78c-pr64k\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.610027 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.610009 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-home\") pod \"test-model-c95db0-kserve-b6dbcd78c-pr64k\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.610104 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.610063 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-tmp-dir\") pod \"test-model-c95db0-kserve-b6dbcd78c-pr64k\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.610104 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.610088 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-model-cache\") pod \"test-model-c95db0-kserve-b6dbcd78c-pr64k\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.611768 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.611751 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-dshm\") pod \"test-model-c95db0-kserve-b6dbcd78c-pr64k\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.612028 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.612013 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-tls-certs\") pod \"test-model-c95db0-kserve-b6dbcd78c-pr64k\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.621387 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.621347 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vqsr\" (UniqueName: \"kubernetes.io/projected/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-kube-api-access-9vqsr\") pod \"test-model-c95db0-kserve-b6dbcd78c-pr64k\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.705390 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.705306 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:49.835039 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:49.835013 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k"] Jul 07 15:10:49.837187 ip-10-0-142-144 kubenswrapper[2583]: W0707 15:10:49.837159 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1b717cd_9c28_4a5b_b6b5_93b9f29bd419.slice/crio-5e0b17a1c102f641aeb3889767128bb878430f2cd04bf9bac8260551ca5fa4d2 WatchSource:0}: Error finding container 5e0b17a1c102f641aeb3889767128bb878430f2cd04bf9bac8260551ca5fa4d2: Status 404 returned error can't find the container with id 5e0b17a1c102f641aeb3889767128bb878430f2cd04bf9bac8260551ca5fa4d2 Jul 07 15:10:50.206922 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:50.206883 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" event={"ID":"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419","Type":"ContainerStarted","Data":"a0613b71adddc18e623e5f3cd5be106530063ee61ac4b21fd138288d5fa7456f"} Jul 07 15:10:50.206922 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:50.206926 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" event={"ID":"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419","Type":"ContainerStarted","Data":"5e0b17a1c102f641aeb3889767128bb878430f2cd04bf9bac8260551ca5fa4d2"} Jul 07 15:10:50.225218 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:50.225163 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" podStartSLOduration=1.225142955 podStartE2EDuration="1.225142955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:10:50.224254113 +0000 UTC m=+2183.135447788" watchObservedRunningTime="2026-07-07 15:10:50.225142955 +0000 UTC m=+2183.136336634" Jul 07 15:10:59.706128 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:59.706097 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:59.706630 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:59.706144 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:10:59.718160 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:10:59.718141 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:11:00.267121 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:00.267094 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:11:09.753816 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:09.753784 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc"] Jul 07 15:11:09.776491 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:09.776464 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:11:10.745946 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:10.745915 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k"] Jul 07 15:11:11.115425 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:11.115332 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:11:11.349454 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:11.349417 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:11:11.450863 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:11.450778 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:11:11.614590 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:11.614558 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:11:16.426914 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:16.426879 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:11:24.773496 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:24.773450 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" podUID="fcaca07b-cb01-48fd-8b34-c8a9099ebce8" containerName="main" containerID="cri-o://1700c3552c4e608c212529cba7f02b303f3ef523342d5111a834a67d843ca7a3" gracePeriod=45 Jul 07 15:11:25.011834 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.011811 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:11:25.090079 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.090015 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6xg7\" (UniqueName: \"kubernetes.io/projected/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-kube-api-access-v6xg7\") pod \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " Jul 07 15:11:25.090079 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.090066 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-dshm\") pod \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " Jul 07 15:11:25.090246 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.090086 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-tls-certs\") pod \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " Jul 07 15:11:25.090246 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.090114 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-home\") pod \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " Jul 07 15:11:25.090246 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.090149 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-tmp-dir\") pod \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " Jul 07 15:11:25.090246 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.090178 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-model-cache\") pod \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\" (UID: \"fcaca07b-cb01-48fd-8b34-c8a9099ebce8\") " Jul 07 15:11:25.090472 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.090437 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-home" (OuterVolumeSpecName: "home") pod "fcaca07b-cb01-48fd-8b34-c8a9099ebce8" (UID: "fcaca07b-cb01-48fd-8b34-c8a9099ebce8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:11:25.090472 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.090460 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fcaca07b-cb01-48fd-8b34-c8a9099ebce8" (UID: "fcaca07b-cb01-48fd-8b34-c8a9099ebce8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:11:25.090587 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.090563 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-model-cache" (OuterVolumeSpecName: "model-cache") pod "fcaca07b-cb01-48fd-8b34-c8a9099ebce8" (UID: "fcaca07b-cb01-48fd-8b34-c8a9099ebce8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:11:25.092288 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.092265 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fcaca07b-cb01-48fd-8b34-c8a9099ebce8" (UID: "fcaca07b-cb01-48fd-8b34-c8a9099ebce8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:11:25.092408 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.092310 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-dshm" (OuterVolumeSpecName: "dshm") pod "fcaca07b-cb01-48fd-8b34-c8a9099ebce8" (UID: "fcaca07b-cb01-48fd-8b34-c8a9099ebce8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:11:25.092408 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.092332 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-kube-api-access-v6xg7" (OuterVolumeSpecName: "kube-api-access-v6xg7") pod "fcaca07b-cb01-48fd-8b34-c8a9099ebce8" (UID: "fcaca07b-cb01-48fd-8b34-c8a9099ebce8"). InnerVolumeSpecName "kube-api-access-v6xg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:11:25.190960 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.190937 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6xg7\" (UniqueName: \"kubernetes.io/projected/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-kube-api-access-v6xg7\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:11:25.190960 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.190959 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-dshm\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:11:25.191117 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.190967 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-tls-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:11:25.191117 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.190976 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-home\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:11:25.191117 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.190984 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-tmp-dir\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:11:25.191117 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.190991 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcaca07b-cb01-48fd-8b34-c8a9099ebce8-model-cache\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:11:25.379298 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.379209 2583 generic.go:356] "Generic (PLEG): container finished" podID="fcaca07b-cb01-48fd-8b34-c8a9099ebce8" containerID="1700c3552c4e608c212529cba7f02b303f3ef523342d5111a834a67d843ca7a3" exitCode=0 Jul 07 15:11:25.379298 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.379254 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" Jul 07 15:11:25.379499 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.379291 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" event={"ID":"fcaca07b-cb01-48fd-8b34-c8a9099ebce8","Type":"ContainerDied","Data":"1700c3552c4e608c212529cba7f02b303f3ef523342d5111a834a67d843ca7a3"} Jul 07 15:11:25.379499 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.379336 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc" event={"ID":"fcaca07b-cb01-48fd-8b34-c8a9099ebce8","Type":"ContainerDied","Data":"63d3b01dfcaf4320e1bb15f2f0b036b63d7f4a27cc5ceb55e6d5cdf60fb18d7b"} Jul 07 15:11:25.379499 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.379354 2583 scope.go:117] "RemoveContainer" containerID="1700c3552c4e608c212529cba7f02b303f3ef523342d5111a834a67d843ca7a3" Jul 07 15:11:25.387935 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.387919 2583 scope.go:117] "RemoveContainer" containerID="1700c3552c4e608c212529cba7f02b303f3ef523342d5111a834a67d843ca7a3" Jul 07 15:11:25.388164 ip-10-0-142-144 kubenswrapper[2583]: E0707 15:11:25.388148 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1700c3552c4e608c212529cba7f02b303f3ef523342d5111a834a67d843ca7a3\": container with ID starting with 1700c3552c4e608c212529cba7f02b303f3ef523342d5111a834a67d843ca7a3 not found: ID does not exist" containerID="1700c3552c4e608c212529cba7f02b303f3ef523342d5111a834a67d843ca7a3" Jul 07 15:11:25.388229 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.388174 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1700c3552c4e608c212529cba7f02b303f3ef523342d5111a834a67d843ca7a3"} err="failed to get container status \"1700c3552c4e608c212529cba7f02b303f3ef523342d5111a834a67d843ca7a3\": rpc error: code = NotFound desc = could not find container \"1700c3552c4e608c212529cba7f02b303f3ef523342d5111a834a67d843ca7a3\": container with ID starting with 1700c3552c4e608c212529cba7f02b303f3ef523342d5111a834a67d843ca7a3 not found: ID does not exist" Jul 07 15:11:25.400498 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.400478 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc"] Jul 07 15:11:25.404450 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.404430 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-45adb8/test-model-45adb8-kserve-5d5fbf788f-7gtpc"] Jul 07 15:11:25.766373 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.766321 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" podUID="d1b717cd-9c28-4a5b-b6b5-93b9f29bd419" containerName="main" containerID="cri-o://a0613b71adddc18e623e5f3cd5be106530063ee61ac4b21fd138288d5fa7456f" gracePeriod=45 Jul 07 15:11:25.792264 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:25.792236 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcaca07b-cb01-48fd-8b34-c8a9099ebce8" path="/var/lib/kubelet/pods/fcaca07b-cb01-48fd-8b34-c8a9099ebce8/volumes" Jul 07 15:11:26.007245 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.007217 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:11:26.098985 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.098901 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-tmp-dir\") pod \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " Jul 07 15:11:26.098985 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.098976 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-home\") pod \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " Jul 07 15:11:26.099177 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.099020 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-tls-certs\") pod \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " Jul 07 15:11:26.099177 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.099047 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vqsr\" (UniqueName: \"kubernetes.io/projected/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-kube-api-access-9vqsr\") pod \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " Jul 07 15:11:26.099177 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.099101 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-dshm\") pod \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " Jul 07 15:11:26.099177 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.099142 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-model-cache\") pod \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\" (UID: \"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419\") " Jul 07 15:11:26.099397 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.099183 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d1b717cd-9c28-4a5b-b6b5-93b9f29bd419" (UID: "d1b717cd-9c28-4a5b-b6b5-93b9f29bd419"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:11:26.099397 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.099264 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-home" (OuterVolumeSpecName: "home") pod "d1b717cd-9c28-4a5b-b6b5-93b9f29bd419" (UID: "d1b717cd-9c28-4a5b-b6b5-93b9f29bd419"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:11:26.099522 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.099465 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-home\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:11:26.099522 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.099470 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-model-cache" (OuterVolumeSpecName: "model-cache") pod "d1b717cd-9c28-4a5b-b6b5-93b9f29bd419" (UID: "d1b717cd-9c28-4a5b-b6b5-93b9f29bd419"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:11:26.099522 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.099485 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-tmp-dir\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:11:26.101252 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.101227 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1b717cd-9c28-4a5b-b6b5-93b9f29bd419" (UID: "d1b717cd-9c28-4a5b-b6b5-93b9f29bd419"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:11:26.101647 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.101618 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-dshm" (OuterVolumeSpecName: "dshm") pod "d1b717cd-9c28-4a5b-b6b5-93b9f29bd419" (UID: "d1b717cd-9c28-4a5b-b6b5-93b9f29bd419"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:11:26.101647 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.101624 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-kube-api-access-9vqsr" (OuterVolumeSpecName: "kube-api-access-9vqsr") pod "d1b717cd-9c28-4a5b-b6b5-93b9f29bd419" (UID: "d1b717cd-9c28-4a5b-b6b5-93b9f29bd419"). InnerVolumeSpecName "kube-api-access-9vqsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:11:26.200933 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.200891 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-tls-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:11:26.200933 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.200922 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vqsr\" (UniqueName: \"kubernetes.io/projected/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-kube-api-access-9vqsr\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:11:26.200933 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.200933 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-dshm\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:11:26.200933 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.200942 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419-model-cache\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:11:26.385984 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.385895 2583 generic.go:356] "Generic (PLEG): container finished" podID="d1b717cd-9c28-4a5b-b6b5-93b9f29bd419" containerID="a0613b71adddc18e623e5f3cd5be106530063ee61ac4b21fd138288d5fa7456f" exitCode=0 Jul 07 15:11:26.385984 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.385937 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" Jul 07 15:11:26.386207 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.385976 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" event={"ID":"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419","Type":"ContainerDied","Data":"a0613b71adddc18e623e5f3cd5be106530063ee61ac4b21fd138288d5fa7456f"} Jul 07 15:11:26.386207 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.386026 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k" event={"ID":"d1b717cd-9c28-4a5b-b6b5-93b9f29bd419","Type":"ContainerDied","Data":"5e0b17a1c102f641aeb3889767128bb878430f2cd04bf9bac8260551ca5fa4d2"} Jul 07 15:11:26.386207 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.386057 2583 scope.go:117] "RemoveContainer" containerID="a0613b71adddc18e623e5f3cd5be106530063ee61ac4b21fd138288d5fa7456f" Jul 07 15:11:26.394895 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.394880 2583 scope.go:117] "RemoveContainer" containerID="a0613b71adddc18e623e5f3cd5be106530063ee61ac4b21fd138288d5fa7456f" Jul 07 15:11:26.395154 ip-10-0-142-144 kubenswrapper[2583]: E0707 15:11:26.395138 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0613b71adddc18e623e5f3cd5be106530063ee61ac4b21fd138288d5fa7456f\": container with ID starting with a0613b71adddc18e623e5f3cd5be106530063ee61ac4b21fd138288d5fa7456f not found: ID does not exist" containerID="a0613b71adddc18e623e5f3cd5be106530063ee61ac4b21fd138288d5fa7456f" Jul 07 15:11:26.395200 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.395162 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0613b71adddc18e623e5f3cd5be106530063ee61ac4b21fd138288d5fa7456f"} err="failed to get container status \"a0613b71adddc18e623e5f3cd5be106530063ee61ac4b21fd138288d5fa7456f\": rpc error: code = NotFound desc = could not find container \"a0613b71adddc18e623e5f3cd5be106530063ee61ac4b21fd138288d5fa7456f\": container with ID starting with a0613b71adddc18e623e5f3cd5be106530063ee61ac4b21fd138288d5fa7456f not found: ID does not exist" Jul 07 15:11:26.408167 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.408143 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k"] Jul 07 15:11:26.412549 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:26.412528 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c95db0/test-model-c95db0-kserve-b6dbcd78c-pr64k"] Jul 07 15:11:27.791128 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:27.791093 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1b717cd-9c28-4a5b-b6b5-93b9f29bd419" path="/var/lib/kubelet/pods/d1b717cd-9c28-4a5b-b6b5-93b9f29bd419/volumes" Jul 07 15:11:53.930757 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:53.930724 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:11:54.228906 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:54.228876 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:11:54.457103 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:54.457068 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:11:54.678378 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:54.678273 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:11:55.050666 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:55.050632 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:11:55.194206 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:55.194169 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:11:55.294506 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:55.294479 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:11:55.486495 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:11:55.486468 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:01.234812 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:01.234775 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:01.353712 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:01.353678 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:01.550622 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:01.550536 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9"] Jul 07 15:12:01.550791 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:01.550769 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" podUID="35fa5688-ba15-4be8-97c7-b786af8587ce" containerName="istio-proxy" containerID="cri-o://02be96c4618a08f414fc866fa9ab5c0e06a4520fff21943d92c95fdeb560eb22" gracePeriod=30 Jul 07 15:12:01.585296 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:01.585266 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:01.698463 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:01.698424 2583 patch_prober.go:28] interesting pod/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 15:12:01.698646 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:01.698501 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" podUID="35fa5688-ba15-4be8-97c7-b786af8587ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 15:12:01.751095 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:01.751064 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:02.137979 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:02.137941 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:02.416922 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:02.416837 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:02.516167 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:02.516132 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:05.811240 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:05.811185 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:05.928945 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:05.928918 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:06.037174 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.037141 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:06.391198 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.391166 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:06.561064 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.561033 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:06.680718 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.680693 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:06.797799 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.797772 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:12:06.833049 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.833014 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-credential-socket\") pod \"35fa5688-ba15-4be8-97c7-b786af8587ce\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " Jul 07 15:12:06.833468 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.833069 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-data\") pod \"35fa5688-ba15-4be8-97c7-b786af8587ce\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " Jul 07 15:12:06.833468 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.833103 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-workload-socket\") pod \"35fa5688-ba15-4be8-97c7-b786af8587ce\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " Jul 07 15:12:06.833468 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.833138 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-token\") pod \"35fa5688-ba15-4be8-97c7-b786af8587ce\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " Jul 07 15:12:06.833468 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.833169 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-envoy\") pod \"35fa5688-ba15-4be8-97c7-b786af8587ce\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " Jul 07 15:12:06.833468 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.833208 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fa5688-ba15-4be8-97c7-b786af8587ce-istiod-ca-cert\") pod \"35fa5688-ba15-4be8-97c7-b786af8587ce\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " Jul 07 15:12:06.833468 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.833238 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-workload-certs\") pod \"35fa5688-ba15-4be8-97c7-b786af8587ce\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " Jul 07 15:12:06.833468 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.833292 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtqcc\" (UniqueName: \"kubernetes.io/projected/35fa5688-ba15-4be8-97c7-b786af8587ce-kube-api-access-xtqcc\") pod \"35fa5688-ba15-4be8-97c7-b786af8587ce\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " Jul 07 15:12:06.833468 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.833314 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "35fa5688-ba15-4be8-97c7-b786af8587ce" (UID: "35fa5688-ba15-4be8-97c7-b786af8587ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:12:06.833468 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.833387 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-podinfo\") pod \"35fa5688-ba15-4be8-97c7-b786af8587ce\" (UID: \"35fa5688-ba15-4be8-97c7-b786af8587ce\") " Jul 07 15:12:06.833468 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.833412 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "35fa5688-ba15-4be8-97c7-b786af8587ce" (UID: "35fa5688-ba15-4be8-97c7-b786af8587ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:12:06.834017 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.833607 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "35fa5688-ba15-4be8-97c7-b786af8587ce" (UID: "35fa5688-ba15-4be8-97c7-b786af8587ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:12:06.834017 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.833652 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-credential-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:12:06.834017 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.833672 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-workload-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:12:06.834017 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.833652 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35fa5688-ba15-4be8-97c7-b786af8587ce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "35fa5688-ba15-4be8-97c7-b786af8587ce" (UID: "35fa5688-ba15-4be8-97c7-b786af8587ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:12:06.834017 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.833973 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-data" (OuterVolumeSpecName: "istio-data") pod "35fa5688-ba15-4be8-97c7-b786af8587ce" (UID: "35fa5688-ba15-4be8-97c7-b786af8587ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:12:06.835579 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.835558 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "35fa5688-ba15-4be8-97c7-b786af8587ce" (UID: "35fa5688-ba15-4be8-97c7-b786af8587ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 15:12:06.835579 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.835569 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fa5688-ba15-4be8-97c7-b786af8587ce-kube-api-access-xtqcc" (OuterVolumeSpecName: "kube-api-access-xtqcc") pod "35fa5688-ba15-4be8-97c7-b786af8587ce" (UID: "35fa5688-ba15-4be8-97c7-b786af8587ce"). InnerVolumeSpecName "kube-api-access-xtqcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:12:06.836110 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.836085 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "35fa5688-ba15-4be8-97c7-b786af8587ce" (UID: "35fa5688-ba15-4be8-97c7-b786af8587ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:12:06.836186 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.836103 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-token" (OuterVolumeSpecName: "istio-token") pod "35fa5688-ba15-4be8-97c7-b786af8587ce" (UID: "35fa5688-ba15-4be8-97c7-b786af8587ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:12:06.935037 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.934976 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-data\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:12:06.935037 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.934998 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-token\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:12:06.935037 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.935010 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-envoy\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:12:06.935037 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.935021 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fa5688-ba15-4be8-97c7-b786af8587ce-istiod-ca-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:12:06.935037 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.935033 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fa5688-ba15-4be8-97c7-b786af8587ce-workload-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:12:06.935247 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.935045 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtqcc\" (UniqueName: \"kubernetes.io/projected/35fa5688-ba15-4be8-97c7-b786af8587ce-kube-api-access-xtqcc\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:12:06.935247 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:06.935057 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fa5688-ba15-4be8-97c7-b786af8587ce-istio-podinfo\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:12:07.586951 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:07.586916 2583 generic.go:356] "Generic (PLEG): container finished" podID="35fa5688-ba15-4be8-97c7-b786af8587ce" containerID="02be96c4618a08f414fc866fa9ab5c0e06a4520fff21943d92c95fdeb560eb22" exitCode=0 Jul 07 15:12:07.587118 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:07.586963 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" Jul 07 15:12:07.587118 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:07.587005 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" event={"ID":"35fa5688-ba15-4be8-97c7-b786af8587ce","Type":"ContainerDied","Data":"02be96c4618a08f414fc866fa9ab5c0e06a4520fff21943d92c95fdeb560eb22"} Jul 07 15:12:07.587118 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:07.587058 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9" event={"ID":"35fa5688-ba15-4be8-97c7-b786af8587ce","Type":"ContainerDied","Data":"5997b7f419984c145a25968ab06a987f3de215d0d2cf27d9d80cb252bceee2d9"} Jul 07 15:12:07.587118 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:07.587083 2583 scope.go:117] "RemoveContainer" containerID="02be96c4618a08f414fc866fa9ab5c0e06a4520fff21943d92c95fdeb560eb22" Jul 07 15:12:07.595905 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:07.595891 2583 scope.go:117] "RemoveContainer" containerID="c3d88811a07a308164ed5e7be40962bf29b45c359ef30db50cc8f19ee9cb89b2" Jul 07 15:12:07.603415 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:07.603400 2583 scope.go:117] "RemoveContainer" containerID="02be96c4618a08f414fc866fa9ab5c0e06a4520fff21943d92c95fdeb560eb22" Jul 07 15:12:07.603645 ip-10-0-142-144 kubenswrapper[2583]: E0707 15:12:07.603628 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02be96c4618a08f414fc866fa9ab5c0e06a4520fff21943d92c95fdeb560eb22\": container with ID starting with 02be96c4618a08f414fc866fa9ab5c0e06a4520fff21943d92c95fdeb560eb22 not found: ID does not exist" containerID="02be96c4618a08f414fc866fa9ab5c0e06a4520fff21943d92c95fdeb560eb22" Jul 07 15:12:07.603690 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:07.603653 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02be96c4618a08f414fc866fa9ab5c0e06a4520fff21943d92c95fdeb560eb22"} err="failed to get container status \"02be96c4618a08f414fc866fa9ab5c0e06a4520fff21943d92c95fdeb560eb22\": rpc error: code = NotFound desc = could not find container \"02be96c4618a08f414fc866fa9ab5c0e06a4520fff21943d92c95fdeb560eb22\": container with ID starting with 02be96c4618a08f414fc866fa9ab5c0e06a4520fff21943d92c95fdeb560eb22 not found: ID does not exist" Jul 07 15:12:07.603690 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:07.603668 2583 scope.go:117] "RemoveContainer" containerID="c3d88811a07a308164ed5e7be40962bf29b45c359ef30db50cc8f19ee9cb89b2" Jul 07 15:12:07.603895 ip-10-0-142-144 kubenswrapper[2583]: E0707 15:12:07.603877 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d88811a07a308164ed5e7be40962bf29b45c359ef30db50cc8f19ee9cb89b2\": container with ID starting with c3d88811a07a308164ed5e7be40962bf29b45c359ef30db50cc8f19ee9cb89b2 not found: ID does not exist" containerID="c3d88811a07a308164ed5e7be40962bf29b45c359ef30db50cc8f19ee9cb89b2" Jul 07 15:12:07.603952 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:07.603904 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d88811a07a308164ed5e7be40962bf29b45c359ef30db50cc8f19ee9cb89b2"} err="failed to get container status \"c3d88811a07a308164ed5e7be40962bf29b45c359ef30db50cc8f19ee9cb89b2\": rpc error: code = NotFound desc = could not find container \"c3d88811a07a308164ed5e7be40962bf29b45c359ef30db50cc8f19ee9cb89b2\": container with ID starting with c3d88811a07a308164ed5e7be40962bf29b45c359ef30db50cc8f19ee9cb89b2 not found: ID does not exist" Jul 07 15:12:07.609490 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:07.609469 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9"] Jul 07 15:12:07.613569 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:07.613553 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c95db0-openshift-default-6c95dd9b89-cbkw9"] Jul 07 15:12:07.791776 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:07.791752 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35fa5688-ba15-4be8-97c7-b786af8587ce" path="/var/lib/kubelet/pods/35fa5688-ba15-4be8-97c7-b786af8587ce/volumes" Jul 07 15:12:10.303139 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:10.303108 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:10.749887 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:10.749858 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:11.073731 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:11.073655 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:11.465311 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:11.465207 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:13.834196 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:13.834156 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:13.939165 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:13.939134 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:14.040077 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:14.040042 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:14.175971 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:14.175888 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:14.508325 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:14.508290 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:14.606881 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:14.606851 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:18.334399 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:18.334342 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:26.532395 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:26.532340 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:26.969001 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:26.968969 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:27.200048 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:27.200006 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:27.299309 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:27.299224 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:27.500497 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:27.500430 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:33.374023 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:33.373984 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:33.473537 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:33.473505 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:33.655080 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:33.654971 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:33.868483 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:33.868452 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:34.053929 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:34.053895 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:34.466431 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:34.466337 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:34.603686 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:34.603615 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:34.702090 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:34.702054 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:34.933778 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:34.933713 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:40.535022 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:40.534977 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:40.668555 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:40.668520 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:40.844614 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:40.844516 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:41.119654 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:41.119546 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:41.571109 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:41.571075 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:41.938091 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:41.938007 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc"] Jul 07 15:12:41.938339 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:41.938325 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35fa5688-ba15-4be8-97c7-b786af8587ce" containerName="istio-proxy" Jul 07 15:12:41.938431 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:41.938341 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="35fa5688-ba15-4be8-97c7-b786af8587ce" containerName="istio-proxy" Jul 07 15:12:41.938431 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:41.938351 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcaca07b-cb01-48fd-8b34-c8a9099ebce8" containerName="main" Jul 07 15:12:41.938431 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:41.938379 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcaca07b-cb01-48fd-8b34-c8a9099ebce8" containerName="main" Jul 07 15:12:41.938431 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:41.938390 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35fa5688-ba15-4be8-97c7-b786af8587ce" containerName="istio-proxy" Jul 07 15:12:41.938431 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:41.938395 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="35fa5688-ba15-4be8-97c7-b786af8587ce" containerName="istio-proxy" Jul 07 15:12:41.938431 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:41.938405 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1b717cd-9c28-4a5b-b6b5-93b9f29bd419" containerName="main" Jul 07 15:12:41.938431 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:41.938410 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1b717cd-9c28-4a5b-b6b5-93b9f29bd419" containerName="main" Jul 07 15:12:41.938786 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:41.938467 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="35fa5688-ba15-4be8-97c7-b786af8587ce" containerName="istio-proxy" Jul 07 15:12:41.938786 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:41.938474 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1b717cd-9c28-4a5b-b6b5-93b9f29bd419" containerName="main" Jul 07 15:12:41.938786 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:41.938482 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcaca07b-cb01-48fd-8b34-c8a9099ebce8" containerName="main" Jul 07 15:12:41.938786 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:41.938489 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="35fa5688-ba15-4be8-97c7-b786af8587ce" containerName="istio-proxy" Jul 07 15:12:41.940169 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:41.940151 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:41.942992 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:41.942961 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-96d90d-openshift-default-dockercfg-rgqdc\"" Jul 07 15:12:41.954505 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:41.954481 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc"] Jul 07 15:12:41.965985 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:41.965960 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:42.063177 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.063146 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:42.131441 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.131403 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-istio-data\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.131605 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.131452 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-workload-socket\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.131605 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.131485 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e2783d-2112-4569-a0c8-674931cc3386-istiod-ca-cert\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.131605 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.131502 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxcst\" (UniqueName: \"kubernetes.io/projected/d7e2783d-2112-4569-a0c8-674931cc3386-kube-api-access-zxcst\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.131605 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.131523 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e2783d-2112-4569-a0c8-674931cc3386-istio-token\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.131842 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.131605 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-workload-certs\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.131842 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.131636 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e2783d-2112-4569-a0c8-674931cc3386-istio-podinfo\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.131842 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.131665 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-istio-envoy\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.131842 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.131683 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-credential-socket\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.232872 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.232840 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e2783d-2112-4569-a0c8-674931cc3386-istiod-ca-cert\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.233028 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.232878 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxcst\" (UniqueName: \"kubernetes.io/projected/d7e2783d-2112-4569-a0c8-674931cc3386-kube-api-access-zxcst\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.233028 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.232902 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e2783d-2112-4569-a0c8-674931cc3386-istio-token\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.233113 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.233074 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-workload-certs\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.233148 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.233114 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e2783d-2112-4569-a0c8-674931cc3386-istio-podinfo\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.233195 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.233152 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-istio-envoy\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.233195 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.233174 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-credential-socket\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.233272 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.233254 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-istio-data\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.233314 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.233291 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-workload-socket\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.233465 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.233441 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-workload-certs\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.233681 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.233655 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-credential-socket\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.233752 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.233668 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-istio-data\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.233752 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.233707 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e2783d-2112-4569-a0c8-674931cc3386-istiod-ca-cert\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.233752 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.233723 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-workload-socket\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.235489 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.235464 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-istio-envoy\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.235662 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.235644 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e2783d-2112-4569-a0c8-674931cc3386-istio-podinfo\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.243096 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.243075 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e2783d-2112-4569-a0c8-674931cc3386-istio-token\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.243303 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.243282 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxcst\" (UniqueName: \"kubernetes.io/projected/d7e2783d-2112-4569-a0c8-674931cc3386-kube-api-access-zxcst\") pod \"e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.251018 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.250999 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:42.381836 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.381810 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc"] Jul 07 15:12:42.383259 ip-10-0-142-144 kubenswrapper[2583]: W0707 15:12:42.383234 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e2783d_2112_4569_a0c8_674931cc3386.slice/crio-91f52b2e2dee0d749516bcc1e68d5f82bdec03506476744160a18a3fab69397b WatchSource:0}: Error finding container 91f52b2e2dee0d749516bcc1e68d5f82bdec03506476744160a18a3fab69397b: Status 404 returned error can't find the container with id 91f52b2e2dee0d749516bcc1e68d5f82bdec03506476744160a18a3fab69397b Jul 07 15:12:42.385403 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.385348 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:42.385493 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.385441 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:42.385493 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.385468 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:42.754643 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.754607 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" event={"ID":"d7e2783d-2112-4569-a0c8-674931cc3386","Type":"ContainerStarted","Data":"f8b4c2799a829f9e27da74ce550e51e88fd8d6740b5141847aeb796ab1ce8adf"} Jul 07 15:12:42.755138 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.754667 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" event={"ID":"d7e2783d-2112-4569-a0c8-674931cc3386","Type":"ContainerStarted","Data":"91f52b2e2dee0d749516bcc1e68d5f82bdec03506476744160a18a3fab69397b"} Jul 07 15:12:42.774048 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:42.773997 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" podStartSLOduration=1.773977071 podStartE2EDuration="1.773977071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:12:42.773193749 +0000 UTC m=+2295.684387426" watchObservedRunningTime="2026-07-07 15:12:42.773977071 +0000 UTC m=+2295.685170750" Jul 07 15:12:43.251440 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:43.251410 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:43.256394 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:43.256370 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:43.760322 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:43.760285 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:43.761312 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:43.761292 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:12:45.818506 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:45.818472 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:45.914299 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:45.914247 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:45.985563 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:45.985526 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:46.276289 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:46.276257 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:46.829676 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:46.829646 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:46.929465 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:46.929433 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:50.445618 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:50.445581 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:50.819755 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:50.819708 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:51.182532 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:51.182450 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:51.281964 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:51.281917 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:51.505527 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:51.505496 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:54.242948 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:54.242909 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:54.370754 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:54.370719 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:54.468532 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:54.468492 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:54.756616 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:54.756570 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:54.891435 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:54.891396 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:54.988662 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:54.988630 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:58.762323 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:58.762229 2583 patch_prober.go:28] interesting pod/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 07 15:12:58.762323 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:58.762294 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" podUID="d7e2783d-2112-4569-a0c8-674931cc3386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 07 15:12:58.829812 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:58.829778 2583 generic.go:356] "Generic (PLEG): container finished" podID="d7e2783d-2112-4569-a0c8-674931cc3386" containerID="f8b4c2799a829f9e27da74ce550e51e88fd8d6740b5141847aeb796ab1ce8adf" exitCode=0 Jul 07 15:12:58.829973 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:58.829852 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" event={"ID":"d7e2783d-2112-4569-a0c8-674931cc3386","Type":"ContainerDied","Data":"f8b4c2799a829f9e27da74ce550e51e88fd8d6740b5141847aeb796ab1ce8adf"} Jul 07 15:12:58.830255 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:58.830239 2583 scope.go:117] "RemoveContainer" containerID="f8b4c2799a829f9e27da74ce550e51e88fd8d6740b5141847aeb796ab1ce8adf" Jul 07 15:12:58.831252 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:58.831227 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:58.831326 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:58.831291 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:58.831326 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:58.831322 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:12:58.840316 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:58.840292 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:12:59.840552 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:12:59.840511 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" event={"ID":"d7e2783d-2112-4569-a0c8-674931cc3386","Type":"ContainerStarted","Data":"8a5a41ada92a41673e99cae80eb2da7f045d17495b64a9ed2469f74e2b11ea7b"} Jul 07 15:13:00.252182 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:00.252142 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:13:00.252586 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:00.252558 2583 patch_prober.go:28] interesting pod/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 07 15:13:00.252775 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:00.252745 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" podUID="d7e2783d-2112-4569-a0c8-674931cc3386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 07 15:13:01.251821 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:01.251779 2583 patch_prober.go:28] interesting pod/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 07 15:13:01.252456 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:01.252424 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" podUID="d7e2783d-2112-4569-a0c8-674931cc3386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 07 15:13:02.251547 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:02.251503 2583 patch_prober.go:28] interesting pod/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 07 15:13:02.251779 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:02.251584 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" podUID="d7e2783d-2112-4569-a0c8-674931cc3386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 07 15:13:03.252284 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:03.252236 2583 patch_prober.go:28] interesting pod/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 07 15:13:03.252748 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:03.252306 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" podUID="d7e2783d-2112-4569-a0c8-674931cc3386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 07 15:13:04.255263 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:04.255234 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:13:04.255774 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:04.255560 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:13:04.256171 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:04.256151 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:13:07.578950 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:07.578897 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:13:07.679224 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:07.679190 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:13:16.535149 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:16.535113 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:13:16.630703 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:16.630672 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:13:27.664306 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.664224 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm"] Jul 07 15:13:27.667450 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.667432 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:27.669933 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.669913 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-96d90d\"/\"openshift-service-ca.crt\"" Jul 07 15:13:27.670835 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.670814 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-96d90d\"/\"default-dockercfg-x2md9\"" Jul 07 15:13:27.670938 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.670832 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-96d90d\"/\"auth-test-model-96d90d-kserve-self-signed-certs\"" Jul 07 15:13:27.670938 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.670856 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-96d90d\"/\"kube-root-ca.crt\"" Jul 07 15:13:27.679340 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.679321 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm"] Jul 07 15:13:27.700241 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.700195 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:13:27.817273 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.817242 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgtcj\" (UniqueName: \"kubernetes.io/projected/078e551a-3ac5-4f97-a19f-33b31768b69d-kube-api-access-cgtcj\") pod \"auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:27.817456 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.817289 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-tmp-dir\") pod \"auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:27.817456 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.817386 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-model-cache\") pod \"auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:27.817456 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.817412 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078e551a-3ac5-4f97-a19f-33b31768b69d-tls-certs\") pod \"auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:27.817456 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.817433 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-home\") pod \"auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:27.817456 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.817451 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-dshm\") pod \"auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:27.918938 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.918840 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-model-cache\") pod \"auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:27.918938 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.918882 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078e551a-3ac5-4f97-a19f-33b31768b69d-tls-certs\") pod \"auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:27.919228 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.919092 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-home\") pod \"auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:27.919228 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.919128 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-dshm\") pod \"auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:27.919228 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.919178 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtcj\" (UniqueName: \"kubernetes.io/projected/078e551a-3ac5-4f97-a19f-33b31768b69d-kube-api-access-cgtcj\") pod \"auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:27.919429 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.919240 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-tmp-dir\") pod \"auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:27.919429 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.919246 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-model-cache\") pod \"auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:27.919548 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.919461 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-home\") pod \"auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:27.919713 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.919690 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-tmp-dir\") pod \"auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:27.921228 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.921201 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-dshm\") pod \"auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:27.921671 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.921650 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078e551a-3ac5-4f97-a19f-33b31768b69d-tls-certs\") pod \"auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:27.930122 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.930098 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgtcj\" (UniqueName: \"kubernetes.io/projected/078e551a-3ac5-4f97-a19f-33b31768b69d-kube-api-access-cgtcj\") pod \"auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:27.976520 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:27.976496 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:28.108568 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:28.108539 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm"] Jul 07 15:13:28.111707 ip-10-0-142-144 kubenswrapper[2583]: W0707 15:13:28.111678 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod078e551a_3ac5_4f97_a19f_33b31768b69d.slice/crio-94021c8835b1b9e649cede0383a9ff1dc5bc8b69d3d9266582177d535f52d0e3 WatchSource:0}: Error finding container 94021c8835b1b9e649cede0383a9ff1dc5bc8b69d3d9266582177d535f52d0e3: Status 404 returned error can't find the container with id 94021c8835b1b9e649cede0383a9ff1dc5bc8b69d3d9266582177d535f52d0e3 Jul 07 15:13:28.979274 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:28.979242 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" event={"ID":"078e551a-3ac5-4f97-a19f-33b31768b69d","Type":"ContainerStarted","Data":"42ecb957f5a74ca9764c42a0acf5ec0aa323f2454db1f96c725cbf99d4bf0270"} Jul 07 15:13:28.979747 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:28.979281 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" event={"ID":"078e551a-3ac5-4f97-a19f-33b31768b69d","Type":"ContainerStarted","Data":"94021c8835b1b9e649cede0383a9ff1dc5bc8b69d3d9266582177d535f52d0e3"} Jul 07 15:13:28.999461 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:28.999382 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" podStartSLOduration=1.9993448470000001 podStartE2EDuration="1.999344847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:13:28.998414449 +0000 UTC m=+2341.909608127" watchObservedRunningTime="2026-07-07 15:13:28.999344847 +0000 UTC m=+2341.910538526" Jul 07 15:13:37.977273 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:37.977234 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:37.977273 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:37.977284 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:37.989961 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:37.989939 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:38.034025 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:38.033999 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:13:49.373341 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.373308 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4"] Jul 07 15:13:49.376653 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.376632 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.379205 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.379188 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-17f250\"/\"default-dockercfg-rml5k\"" Jul 07 15:13:49.379905 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.379883 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-17f250\"/\"openshift-service-ca.crt\"" Jul 07 15:13:49.380011 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.379922 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-17f250\"/\"kube-root-ca.crt\"" Jul 07 15:13:49.380011 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.379969 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-17f250\"/\"auth-test-model-17f250-kserve-self-signed-certs\"" Jul 07 15:13:49.386153 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.386128 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4"] Jul 07 15:13:49.466725 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.466688 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:13:49.479009 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.478980 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-home\") pod \"auth-test-model-17f250-kserve-774459fb5f-ckgl4\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.479112 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.479013 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-model-cache\") pod \"auth-test-model-17f250-kserve-774459fb5f-ckgl4\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.479178 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.479119 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-dshm\") pod \"auth-test-model-17f250-kserve-774459fb5f-ckgl4\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.479178 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.479147 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65af52cb-4051-48ad-9dea-1a0016c977b2-tls-certs\") pod \"auth-test-model-17f250-kserve-774459fb5f-ckgl4\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.479286 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.479222 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9csm\" (UniqueName: \"kubernetes.io/projected/65af52cb-4051-48ad-9dea-1a0016c977b2-kube-api-access-m9csm\") pod \"auth-test-model-17f250-kserve-774459fb5f-ckgl4\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.479286 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.479269 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-tmp-dir\") pod \"auth-test-model-17f250-kserve-774459fb5f-ckgl4\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.580262 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.580231 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-dshm\") pod \"auth-test-model-17f250-kserve-774459fb5f-ckgl4\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.580403 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.580271 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65af52cb-4051-48ad-9dea-1a0016c977b2-tls-certs\") pod \"auth-test-model-17f250-kserve-774459fb5f-ckgl4\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.580460 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.580404 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9csm\" (UniqueName: \"kubernetes.io/projected/65af52cb-4051-48ad-9dea-1a0016c977b2-kube-api-access-m9csm\") pod \"auth-test-model-17f250-kserve-774459fb5f-ckgl4\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.580460 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.580448 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-tmp-dir\") pod \"auth-test-model-17f250-kserve-774459fb5f-ckgl4\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.580537 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.580484 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-home\") pod \"auth-test-model-17f250-kserve-774459fb5f-ckgl4\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.580537 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.580507 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-model-cache\") pod \"auth-test-model-17f250-kserve-774459fb5f-ckgl4\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.580795 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.580770 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-tmp-dir\") pod \"auth-test-model-17f250-kserve-774459fb5f-ckgl4\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.580907 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.580823 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-home\") pod \"auth-test-model-17f250-kserve-774459fb5f-ckgl4\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.580907 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.580863 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-model-cache\") pod \"auth-test-model-17f250-kserve-774459fb5f-ckgl4\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.582468 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.582448 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-dshm\") pod \"auth-test-model-17f250-kserve-774459fb5f-ckgl4\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.582727 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.582711 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65af52cb-4051-48ad-9dea-1a0016c977b2-tls-certs\") pod \"auth-test-model-17f250-kserve-774459fb5f-ckgl4\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.589508 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.589486 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9csm\" (UniqueName: \"kubernetes.io/projected/65af52cb-4051-48ad-9dea-1a0016c977b2-kube-api-access-m9csm\") pod \"auth-test-model-17f250-kserve-774459fb5f-ckgl4\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.687528 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.687468 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:49.834264 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:49.834241 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4"] Jul 07 15:13:49.835737 ip-10-0-142-144 kubenswrapper[2583]: W0707 15:13:49.835704 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65af52cb_4051_48ad_9dea_1a0016c977b2.slice/crio-48fe2ddf6528419d16dc80165209e8fa75c11f46f78f15677eb3ae7c879dbbb3 WatchSource:0}: Error finding container 48fe2ddf6528419d16dc80165209e8fa75c11f46f78f15677eb3ae7c879dbbb3: Status 404 returned error can't find the container with id 48fe2ddf6528419d16dc80165209e8fa75c11f46f78f15677eb3ae7c879dbbb3 Jul 07 15:13:50.080692 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:50.080657 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" event={"ID":"65af52cb-4051-48ad-9dea-1a0016c977b2","Type":"ContainerStarted","Data":"e6aff567d0ea347a15b1e214a51670a334e0ad52ee05bd6769df4415a8766d89"} Jul 07 15:13:50.080852 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:50.080702 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" event={"ID":"65af52cb-4051-48ad-9dea-1a0016c977b2","Type":"ContainerStarted","Data":"48fe2ddf6528419d16dc80165209e8fa75c11f46f78f15677eb3ae7c879dbbb3"} Jul 07 15:13:50.100983 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:50.100937 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" podStartSLOduration=1.100923643 podStartE2EDuration="1.100923643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:13:50.100123559 +0000 UTC m=+2363.011317239" watchObservedRunningTime="2026-07-07 15:13:50.100923643 +0000 UTC m=+2363.012117320" Jul 07 15:13:59.688051 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:59.688017 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:59.688051 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:59.688058 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:13:59.699940 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:13:59.699917 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:14:00.138816 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:14:00.138791 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:14:27.744250 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:14:27.744216 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 15:14:27.752165 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:14:27.752145 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 15:14:43.646831 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:14:43.646783 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:14:43.715624 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:14:43.715596 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:14:58.182822 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:14:58.182772 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:15:26.366378 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:26.366302 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:15:26.504705 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:26.504674 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:15:31.767862 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:31.767826 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:15:42.097845 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:42.097808 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm"] Jul 07 15:15:42.122173 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:42.122144 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:15:43.066662 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:43.066624 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4"] Jul 07 15:15:43.089212 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:43.089178 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:15:45.364832 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.364798 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl"] Jul 07 15:15:45.367870 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.367853 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.370670 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.370648 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-96d90d\"/\"sub-test-model-96d90d-kserve-self-signed-certs\"" Jul 07 15:15:45.378486 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.378465 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl"] Jul 07 15:15:45.391804 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.391779 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:15:45.518238 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.518206 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b023e403-093c-402a-bb66-752ee5221bfd-tls-certs\") pod \"sub-test-model-96d90d-kserve-69cf46fd4c-n9snl\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.518410 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.518246 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-dshm\") pod \"sub-test-model-96d90d-kserve-69cf46fd4c-n9snl\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.518410 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.518265 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkqxz\" (UniqueName: \"kubernetes.io/projected/b023e403-093c-402a-bb66-752ee5221bfd-kube-api-access-qkqxz\") pod \"sub-test-model-96d90d-kserve-69cf46fd4c-n9snl\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.518410 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.518339 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-model-cache\") pod \"sub-test-model-96d90d-kserve-69cf46fd4c-n9snl\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.518534 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.518425 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-home\") pod \"sub-test-model-96d90d-kserve-69cf46fd4c-n9snl\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.518534 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.518453 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-tmp-dir\") pod \"sub-test-model-96d90d-kserve-69cf46fd4c-n9snl\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.619513 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.619435 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b023e403-093c-402a-bb66-752ee5221bfd-tls-certs\") pod \"sub-test-model-96d90d-kserve-69cf46fd4c-n9snl\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.619513 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.619479 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-dshm\") pod \"sub-test-model-96d90d-kserve-69cf46fd4c-n9snl\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.619513 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.619504 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkqxz\" (UniqueName: \"kubernetes.io/projected/b023e403-093c-402a-bb66-752ee5221bfd-kube-api-access-qkqxz\") pod \"sub-test-model-96d90d-kserve-69cf46fd4c-n9snl\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.619765 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.619559 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-model-cache\") pod \"sub-test-model-96d90d-kserve-69cf46fd4c-n9snl\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.619765 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.619596 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-home\") pod \"sub-test-model-96d90d-kserve-69cf46fd4c-n9snl\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.619765 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.619619 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-tmp-dir\") pod \"sub-test-model-96d90d-kserve-69cf46fd4c-n9snl\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.620105 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.620069 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-tmp-dir\") pod \"sub-test-model-96d90d-kserve-69cf46fd4c-n9snl\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.620105 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.620092 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-model-cache\") pod \"sub-test-model-96d90d-kserve-69cf46fd4c-n9snl\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.620253 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.620212 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-home\") pod \"sub-test-model-96d90d-kserve-69cf46fd4c-n9snl\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.622501 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.622328 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-dshm\") pod \"sub-test-model-96d90d-kserve-69cf46fd4c-n9snl\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.622942 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.622912 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b023e403-093c-402a-bb66-752ee5221bfd-tls-certs\") pod \"sub-test-model-96d90d-kserve-69cf46fd4c-n9snl\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.628456 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.628433 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkqxz\" (UniqueName: \"kubernetes.io/projected/b023e403-093c-402a-bb66-752ee5221bfd-kube-api-access-qkqxz\") pod \"sub-test-model-96d90d-kserve-69cf46fd4c-n9snl\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.678329 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.678305 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:45.808386 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:45.808327 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl"] Jul 07 15:15:45.810683 ip-10-0-142-144 kubenswrapper[2583]: W0707 15:15:45.810649 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb023e403_093c_402a_bb66_752ee5221bfd.slice/crio-1f12fbb0993fd086e2b23100b1d46a9be50b89b3c2317f82decf0129f995cf48 WatchSource:0}: Error finding container 1f12fbb0993fd086e2b23100b1d46a9be50b89b3c2317f82decf0129f995cf48: Status 404 returned error can't find the container with id 1f12fbb0993fd086e2b23100b1d46a9be50b89b3c2317f82decf0129f995cf48 Jul 07 15:15:46.647096 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:46.647067 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" event={"ID":"b023e403-093c-402a-bb66-752ee5221bfd","Type":"ContainerStarted","Data":"6595209af3e570923985ac1f7a48265d3db88376fb542a7dac500ff165101299"} Jul 07 15:15:46.647096 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:46.647107 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" event={"ID":"b023e403-093c-402a-bb66-752ee5221bfd","Type":"ContainerStarted","Data":"1f12fbb0993fd086e2b23100b1d46a9be50b89b3c2317f82decf0129f995cf48"} Jul 07 15:15:46.669484 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:46.669441 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" podStartSLOduration=1.6694270279999999 podStartE2EDuration="1.669427028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:15:46.666590574 +0000 UTC m=+2479.577784263" watchObservedRunningTime="2026-07-07 15:15:46.669427028 +0000 UTC m=+2479.580620706" Jul 07 15:15:55.679441 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:55.679402 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:55.679878 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:55.679460 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:55.692892 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:55.692864 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:55.704189 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:55.704165 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:15:57.117443 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.117380 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" podUID="078e551a-3ac5-4f97-a19f-33b31768b69d" containerName="main" containerID="cri-o://42ecb957f5a74ca9764c42a0acf5ec0aa323f2454db1f96c725cbf99d4bf0270" gracePeriod=45 Jul 07 15:15:57.358022 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.357991 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:15:57.408521 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.408446 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgtcj\" (UniqueName: \"kubernetes.io/projected/078e551a-3ac5-4f97-a19f-33b31768b69d-kube-api-access-cgtcj\") pod \"078e551a-3ac5-4f97-a19f-33b31768b69d\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " Jul 07 15:15:57.408521 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.408514 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-tmp-dir\") pod \"078e551a-3ac5-4f97-a19f-33b31768b69d\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " Jul 07 15:15:57.408731 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.408538 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-home\") pod \"078e551a-3ac5-4f97-a19f-33b31768b69d\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " Jul 07 15:15:57.408731 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.408707 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078e551a-3ac5-4f97-a19f-33b31768b69d-tls-certs\") pod \"078e551a-3ac5-4f97-a19f-33b31768b69d\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " Jul 07 15:15:57.408851 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.408742 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-home" (OuterVolumeSpecName: "home") pod "078e551a-3ac5-4f97-a19f-33b31768b69d" (UID: "078e551a-3ac5-4f97-a19f-33b31768b69d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:15:57.408851 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.408760 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-model-cache\") pod \"078e551a-3ac5-4f97-a19f-33b31768b69d\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " Jul 07 15:15:57.408851 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.408757 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "078e551a-3ac5-4f97-a19f-33b31768b69d" (UID: "078e551a-3ac5-4f97-a19f-33b31768b69d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:15:57.408851 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.408829 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-dshm\") pod \"078e551a-3ac5-4f97-a19f-33b31768b69d\" (UID: \"078e551a-3ac5-4f97-a19f-33b31768b69d\") " Jul 07 15:15:57.409013 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.408905 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-model-cache" (OuterVolumeSpecName: "model-cache") pod "078e551a-3ac5-4f97-a19f-33b31768b69d" (UID: "078e551a-3ac5-4f97-a19f-33b31768b69d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:15:57.409158 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.409134 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-home\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:15:57.409227 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.409164 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-model-cache\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:15:57.409227 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.409178 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-tmp-dir\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:15:57.410763 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.410738 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078e551a-3ac5-4f97-a19f-33b31768b69d-kube-api-access-cgtcj" (OuterVolumeSpecName: "kube-api-access-cgtcj") pod "078e551a-3ac5-4f97-a19f-33b31768b69d" (UID: "078e551a-3ac5-4f97-a19f-33b31768b69d"). InnerVolumeSpecName "kube-api-access-cgtcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:15:57.410858 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.410776 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-dshm" (OuterVolumeSpecName: "dshm") pod "078e551a-3ac5-4f97-a19f-33b31768b69d" (UID: "078e551a-3ac5-4f97-a19f-33b31768b69d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:15:57.410858 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.410839 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078e551a-3ac5-4f97-a19f-33b31768b69d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "078e551a-3ac5-4f97-a19f-33b31768b69d" (UID: "078e551a-3ac5-4f97-a19f-33b31768b69d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:15:57.510143 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.510119 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/078e551a-3ac5-4f97-a19f-33b31768b69d-tls-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:15:57.510143 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.510140 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/078e551a-3ac5-4f97-a19f-33b31768b69d-dshm\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:15:57.510286 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.510150 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgtcj\" (UniqueName: \"kubernetes.io/projected/078e551a-3ac5-4f97-a19f-33b31768b69d-kube-api-access-cgtcj\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:15:57.704591 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.704527 2583 generic.go:356] "Generic (PLEG): container finished" podID="078e551a-3ac5-4f97-a19f-33b31768b69d" containerID="42ecb957f5a74ca9764c42a0acf5ec0aa323f2454db1f96c725cbf99d4bf0270" exitCode=0 Jul 07 15:15:57.704591 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.704574 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" Jul 07 15:15:57.704744 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.704613 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" event={"ID":"078e551a-3ac5-4f97-a19f-33b31768b69d","Type":"ContainerDied","Data":"42ecb957f5a74ca9764c42a0acf5ec0aa323f2454db1f96c725cbf99d4bf0270"} Jul 07 15:15:57.704744 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.704658 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm" event={"ID":"078e551a-3ac5-4f97-a19f-33b31768b69d","Type":"ContainerDied","Data":"94021c8835b1b9e649cede0383a9ff1dc5bc8b69d3d9266582177d535f52d0e3"} Jul 07 15:15:57.704744 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.704677 2583 scope.go:117] "RemoveContainer" containerID="42ecb957f5a74ca9764c42a0acf5ec0aa323f2454db1f96c725cbf99d4bf0270" Jul 07 15:15:57.714446 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.714429 2583 scope.go:117] "RemoveContainer" containerID="42ecb957f5a74ca9764c42a0acf5ec0aa323f2454db1f96c725cbf99d4bf0270" Jul 07 15:15:57.714735 ip-10-0-142-144 kubenswrapper[2583]: E0707 15:15:57.714716 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ecb957f5a74ca9764c42a0acf5ec0aa323f2454db1f96c725cbf99d4bf0270\": container with ID starting with 42ecb957f5a74ca9764c42a0acf5ec0aa323f2454db1f96c725cbf99d4bf0270 not found: ID does not exist" containerID="42ecb957f5a74ca9764c42a0acf5ec0aa323f2454db1f96c725cbf99d4bf0270" Jul 07 15:15:57.714777 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.714744 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ecb957f5a74ca9764c42a0acf5ec0aa323f2454db1f96c725cbf99d4bf0270"} err="failed to get container status \"42ecb957f5a74ca9764c42a0acf5ec0aa323f2454db1f96c725cbf99d4bf0270\": rpc error: code = NotFound desc = could not find container \"42ecb957f5a74ca9764c42a0acf5ec0aa323f2454db1f96c725cbf99d4bf0270\": container with ID starting with 42ecb957f5a74ca9764c42a0acf5ec0aa323f2454db1f96c725cbf99d4bf0270 not found: ID does not exist" Jul 07 15:15:57.726118 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.726091 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm"] Jul 07 15:15:57.731530 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.731508 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-96d90d/auth-test-model-96d90d-kserve-5cbcd4b855-mxcmm"] Jul 07 15:15:57.792972 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:57.792946 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="078e551a-3ac5-4f97-a19f-33b31768b69d" path="/var/lib/kubelet/pods/078e551a-3ac5-4f97-a19f-33b31768b69d/volumes" Jul 07 15:15:58.086482 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.086444 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" podUID="65af52cb-4051-48ad-9dea-1a0016c977b2" containerName="main" containerID="cri-o://e6aff567d0ea347a15b1e214a51670a334e0ad52ee05bd6769df4415a8766d89" gracePeriod=45 Jul 07 15:15:58.328974 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.328953 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:15:58.419311 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.419249 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-home\") pod \"65af52cb-4051-48ad-9dea-1a0016c977b2\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " Jul 07 15:15:58.419311 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.419283 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-tmp-dir\") pod \"65af52cb-4051-48ad-9dea-1a0016c977b2\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " Jul 07 15:15:58.419477 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.419337 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9csm\" (UniqueName: \"kubernetes.io/projected/65af52cb-4051-48ad-9dea-1a0016c977b2-kube-api-access-m9csm\") pod \"65af52cb-4051-48ad-9dea-1a0016c977b2\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " Jul 07 15:15:58.419477 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.419383 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65af52cb-4051-48ad-9dea-1a0016c977b2-tls-certs\") pod \"65af52cb-4051-48ad-9dea-1a0016c977b2\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " Jul 07 15:15:58.419477 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.419453 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-dshm\") pod \"65af52cb-4051-48ad-9dea-1a0016c977b2\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " Jul 07 15:15:58.419641 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.419479 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-model-cache\") pod \"65af52cb-4051-48ad-9dea-1a0016c977b2\" (UID: \"65af52cb-4051-48ad-9dea-1a0016c977b2\") " Jul 07 15:15:58.419641 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.419551 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-home" (OuterVolumeSpecName: "home") pod "65af52cb-4051-48ad-9dea-1a0016c977b2" (UID: "65af52cb-4051-48ad-9dea-1a0016c977b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:15:58.419641 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.419608 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65af52cb-4051-48ad-9dea-1a0016c977b2" (UID: "65af52cb-4051-48ad-9dea-1a0016c977b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:15:58.419810 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.419779 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-home\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:15:58.419810 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.419793 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-tmp-dir\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:15:58.419891 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.419815 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "65af52cb-4051-48ad-9dea-1a0016c977b2" (UID: "65af52cb-4051-48ad-9dea-1a0016c977b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:15:58.421483 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.421456 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-dshm" (OuterVolumeSpecName: "dshm") pod "65af52cb-4051-48ad-9dea-1a0016c977b2" (UID: "65af52cb-4051-48ad-9dea-1a0016c977b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:15:58.421580 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.421521 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65af52cb-4051-48ad-9dea-1a0016c977b2-kube-api-access-m9csm" (OuterVolumeSpecName: "kube-api-access-m9csm") pod "65af52cb-4051-48ad-9dea-1a0016c977b2" (UID: "65af52cb-4051-48ad-9dea-1a0016c977b2"). InnerVolumeSpecName "kube-api-access-m9csm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:15:58.421580 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.421521 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65af52cb-4051-48ad-9dea-1a0016c977b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65af52cb-4051-48ad-9dea-1a0016c977b2" (UID: "65af52cb-4051-48ad-9dea-1a0016c977b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:15:58.520296 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.520268 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9csm\" (UniqueName: \"kubernetes.io/projected/65af52cb-4051-48ad-9dea-1a0016c977b2-kube-api-access-m9csm\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:15:58.520296 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.520293 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65af52cb-4051-48ad-9dea-1a0016c977b2-tls-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:15:58.520458 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.520307 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-dshm\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:15:58.520458 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.520318 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65af52cb-4051-48ad-9dea-1a0016c977b2-model-cache\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:15:58.712069 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.712010 2583 generic.go:356] "Generic (PLEG): container finished" podID="65af52cb-4051-48ad-9dea-1a0016c977b2" containerID="e6aff567d0ea347a15b1e214a51670a334e0ad52ee05bd6769df4415a8766d89" exitCode=0 Jul 07 15:15:58.712069 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.712051 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" Jul 07 15:15:58.712204 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.712093 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" event={"ID":"65af52cb-4051-48ad-9dea-1a0016c977b2","Type":"ContainerDied","Data":"e6aff567d0ea347a15b1e214a51670a334e0ad52ee05bd6769df4415a8766d89"} Jul 07 15:15:58.712204 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.712136 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4" event={"ID":"65af52cb-4051-48ad-9dea-1a0016c977b2","Type":"ContainerDied","Data":"48fe2ddf6528419d16dc80165209e8fa75c11f46f78f15677eb3ae7c879dbbb3"} Jul 07 15:15:58.712204 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.712159 2583 scope.go:117] "RemoveContainer" containerID="e6aff567d0ea347a15b1e214a51670a334e0ad52ee05bd6769df4415a8766d89" Jul 07 15:15:58.720629 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.720613 2583 scope.go:117] "RemoveContainer" containerID="e6aff567d0ea347a15b1e214a51670a334e0ad52ee05bd6769df4415a8766d89" Jul 07 15:15:58.720921 ip-10-0-142-144 kubenswrapper[2583]: E0707 15:15:58.720903 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6aff567d0ea347a15b1e214a51670a334e0ad52ee05bd6769df4415a8766d89\": container with ID starting with e6aff567d0ea347a15b1e214a51670a334e0ad52ee05bd6769df4415a8766d89 not found: ID does not exist" containerID="e6aff567d0ea347a15b1e214a51670a334e0ad52ee05bd6769df4415a8766d89" Jul 07 15:15:58.720988 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.720929 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6aff567d0ea347a15b1e214a51670a334e0ad52ee05bd6769df4415a8766d89"} err="failed to get container status \"e6aff567d0ea347a15b1e214a51670a334e0ad52ee05bd6769df4415a8766d89\": rpc error: code = NotFound desc = could not find container \"e6aff567d0ea347a15b1e214a51670a334e0ad52ee05bd6769df4415a8766d89\": container with ID starting with e6aff567d0ea347a15b1e214a51670a334e0ad52ee05bd6769df4415a8766d89 not found: ID does not exist" Jul 07 15:15:58.732200 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.732180 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4"] Jul 07 15:15:58.736053 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:58.736035 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-17f250/auth-test-model-17f250-kserve-774459fb5f-ckgl4"] Jul 07 15:15:59.791767 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:15:59.791718 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65af52cb-4051-48ad-9dea-1a0016c977b2" path="/var/lib/kubelet/pods/65af52cb-4051-48ad-9dea-1a0016c977b2/volumes" Jul 07 15:16:05.685556 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.685521 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf"] Jul 07 15:16:05.685921 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.685825 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65af52cb-4051-48ad-9dea-1a0016c977b2" containerName="main" Jul 07 15:16:05.685921 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.685835 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="65af52cb-4051-48ad-9dea-1a0016c977b2" containerName="main" Jul 07 15:16:05.685921 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.685849 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="078e551a-3ac5-4f97-a19f-33b31768b69d" containerName="main" Jul 07 15:16:05.685921 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.685854 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="078e551a-3ac5-4f97-a19f-33b31768b69d" containerName="main" Jul 07 15:16:05.685921 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.685917 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="65af52cb-4051-48ad-9dea-1a0016c977b2" containerName="main" Jul 07 15:16:05.686094 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.685929 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="078e551a-3ac5-4f97-a19f-33b31768b69d" containerName="main" Jul 07 15:16:05.688746 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.688726 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:05.693235 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.693218 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-17f250\"/\"default-dockercfg-rml5k\"" Jul 07 15:16:05.693330 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.693286 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-17f250\"/\"sub-test-model-17f250-kserve-self-signed-certs\"" Jul 07 15:16:05.693649 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.693632 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-17f250\"/\"openshift-service-ca.crt\"" Jul 07 15:16:05.694022 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.694005 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-17f250\"/\"kube-root-ca.crt\"" Jul 07 15:16:05.706802 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.706781 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf"] Jul 07 15:16:05.770674 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.770631 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-home\") pod \"sub-test-model-17f250-kserve-598975c465-6z5hf\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:05.770674 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.770662 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/530839b6-d1d6-45cd-8235-8e12c7307e74-tls-certs\") pod \"sub-test-model-17f250-kserve-598975c465-6z5hf\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:05.770856 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.770694 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqgs5\" (UniqueName: \"kubernetes.io/projected/530839b6-d1d6-45cd-8235-8e12c7307e74-kube-api-access-gqgs5\") pod \"sub-test-model-17f250-kserve-598975c465-6z5hf\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:05.770856 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.770779 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-model-cache\") pod \"sub-test-model-17f250-kserve-598975c465-6z5hf\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:05.770856 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.770815 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-tmp-dir\") pod \"sub-test-model-17f250-kserve-598975c465-6z5hf\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:05.770964 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.770863 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-dshm\") pod \"sub-test-model-17f250-kserve-598975c465-6z5hf\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:05.787147 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.787116 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:16:05.871443 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.871418 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-home\") pod \"sub-test-model-17f250-kserve-598975c465-6z5hf\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:05.871549 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.871446 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/530839b6-d1d6-45cd-8235-8e12c7307e74-tls-certs\") pod \"sub-test-model-17f250-kserve-598975c465-6z5hf\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:05.871549 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.871494 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgs5\" (UniqueName: \"kubernetes.io/projected/530839b6-d1d6-45cd-8235-8e12c7307e74-kube-api-access-gqgs5\") pod \"sub-test-model-17f250-kserve-598975c465-6z5hf\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:05.871653 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.871559 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-model-cache\") pod \"sub-test-model-17f250-kserve-598975c465-6z5hf\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:05.871653 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.871607 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-tmp-dir\") pod \"sub-test-model-17f250-kserve-598975c465-6z5hf\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:05.871770 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.871673 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-dshm\") pod \"sub-test-model-17f250-kserve-598975c465-6z5hf\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:05.871885 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.871864 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-home\") pod \"sub-test-model-17f250-kserve-598975c465-6z5hf\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:05.872059 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.872017 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-tmp-dir\") pod \"sub-test-model-17f250-kserve-598975c465-6z5hf\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:05.872166 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.872084 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-model-cache\") pod \"sub-test-model-17f250-kserve-598975c465-6z5hf\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:05.873813 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.873796 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-dshm\") pod \"sub-test-model-17f250-kserve-598975c465-6z5hf\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:05.874080 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.874061 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/530839b6-d1d6-45cd-8235-8e12c7307e74-tls-certs\") pod \"sub-test-model-17f250-kserve-598975c465-6z5hf\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:05.881655 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.881635 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqgs5\" (UniqueName: \"kubernetes.io/projected/530839b6-d1d6-45cd-8235-8e12c7307e74-kube-api-access-gqgs5\") pod \"sub-test-model-17f250-kserve-598975c465-6z5hf\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:05.997885 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:05.997856 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:06.130717 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:06.130689 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf"] Jul 07 15:16:06.132379 ip-10-0-142-144 kubenswrapper[2583]: W0707 15:16:06.132339 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod530839b6_d1d6_45cd_8235_8e12c7307e74.slice/crio-180b9fb0c750b29eabf455a15bd66a6bd19966569afab014d4b43b3bebc33078 WatchSource:0}: Error finding container 180b9fb0c750b29eabf455a15bd66a6bd19966569afab014d4b43b3bebc33078: Status 404 returned error can't find the container with id 180b9fb0c750b29eabf455a15bd66a6bd19966569afab014d4b43b3bebc33078 Jul 07 15:16:06.751928 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:06.751898 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" event={"ID":"530839b6-d1d6-45cd-8235-8e12c7307e74","Type":"ContainerStarted","Data":"c58187751f3eb17d710d6c1393572b6ae71a67e40f297fa6897bb18cc4c2dfb4"} Jul 07 15:16:06.752301 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:06.751935 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" event={"ID":"530839b6-d1d6-45cd-8235-8e12c7307e74","Type":"ContainerStarted","Data":"180b9fb0c750b29eabf455a15bd66a6bd19966569afab014d4b43b3bebc33078"} Jul 07 15:16:06.772165 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:06.772117 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" podStartSLOduration=1.772103156 podStartE2EDuration="1.772103156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:16:06.771321138 +0000 UTC m=+2499.682514814" watchObservedRunningTime="2026-07-07 15:16:06.772103156 +0000 UTC m=+2499.683296833" Jul 07 15:16:15.998277 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:15.998247 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:15.998746 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:15.998295 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:16.010095 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:16.010074 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:16.810704 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:16.810676 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:16:41.383716 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:41.383683 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:16:46.412383 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:46.412336 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl"] Jul 07 15:16:46.434023 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:46.433997 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:16:47.380940 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:47.380907 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf"] Jul 07 15:16:47.405858 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:47.405833 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:16:52.463663 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.463630 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r"] Jul 07 15:16:52.466307 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.466288 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.468966 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.468934 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-96d90d\"/\"rate-test-model-96d90d-kserve-self-signed-certs\"" Jul 07 15:16:52.477098 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.477076 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r"] Jul 07 15:16:52.499140 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.499115 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:16:52.624078 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.624046 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-dshm\") pod \"rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.624258 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.624096 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-tmp-dir\") pod \"rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.624258 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.624165 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4kxp\" (UniqueName: \"kubernetes.io/projected/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-kube-api-access-q4kxp\") pod \"rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.624354 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.624250 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-home\") pod \"rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.624354 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.624276 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-tls-certs\") pod \"rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.624354 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.624332 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-model-cache\") pod \"rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.724858 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.724824 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4kxp\" (UniqueName: \"kubernetes.io/projected/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-kube-api-access-q4kxp\") pod \"rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.725029 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.724873 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-home\") pod \"rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.725029 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.724901 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-tls-certs\") pod \"rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.725029 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.724953 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-model-cache\") pod \"rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.725029 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.724984 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-dshm\") pod \"rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.725293 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.725049 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-tmp-dir\") pod \"rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.725454 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.725411 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-home\") pod \"rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.725454 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.725445 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-model-cache\") pod \"rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.725657 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.725617 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-tmp-dir\") pod \"rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.727421 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.727396 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-dshm\") pod \"rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.727551 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.727534 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-tls-certs\") pod \"rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.733187 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.733171 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4kxp\" (UniqueName: \"kubernetes.io/projected/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-kube-api-access-q4kxp\") pod \"rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.775887 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.775864 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:16:52.907817 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.907787 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r"] Jul 07 15:16:52.908637 ip-10-0-142-144 kubenswrapper[2583]: W0707 15:16:52.908611 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23425ba4_3ce5_49fb_a6c4_d2b7d80e90cc.slice/crio-653e1bec6587f0d43138f432f5ae89c57235f490ae15b94055db64cdecb4b049 WatchSource:0}: Error finding container 653e1bec6587f0d43138f432f5ae89c57235f490ae15b94055db64cdecb4b049: Status 404 returned error can't find the container with id 653e1bec6587f0d43138f432f5ae89c57235f490ae15b94055db64cdecb4b049 Jul 07 15:16:52.979379 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.979277 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" event={"ID":"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc","Type":"ContainerStarted","Data":"6ff66d4bc4e82674c741be15967f8e272a724a317c44c7ca57439f886038a4a6"} Jul 07 15:16:52.979379 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:52.979343 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" event={"ID":"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc","Type":"ContainerStarted","Data":"653e1bec6587f0d43138f432f5ae89c57235f490ae15b94055db64cdecb4b049"} Jul 07 15:16:53.001319 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:16:53.001271 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" podStartSLOduration=1.001256962 podStartE2EDuration="1.001256962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:16:52.999289735 +0000 UTC m=+2545.910483424" watchObservedRunningTime="2026-07-07 15:16:53.001256962 +0000 UTC m=+2545.912450645" Jul 07 15:17:01.431191 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.431131 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" podUID="b023e403-093c-402a-bb66-752ee5221bfd" containerName="main" containerID="cri-o://6595209af3e570923985ac1f7a48265d3db88376fb542a7dac500ff165101299" gracePeriod=45 Jul 07 15:17:01.676706 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.676680 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:17:01.797097 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.797064 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b023e403-093c-402a-bb66-752ee5221bfd-tls-certs\") pod \"b023e403-093c-402a-bb66-752ee5221bfd\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " Jul 07 15:17:01.797276 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.797147 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkqxz\" (UniqueName: \"kubernetes.io/projected/b023e403-093c-402a-bb66-752ee5221bfd-kube-api-access-qkqxz\") pod \"b023e403-093c-402a-bb66-752ee5221bfd\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " Jul 07 15:17:01.797276 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.797217 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-tmp-dir\") pod \"b023e403-093c-402a-bb66-752ee5221bfd\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " Jul 07 15:17:01.797276 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.797260 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-dshm\") pod \"b023e403-093c-402a-bb66-752ee5221bfd\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " Jul 07 15:17:01.797487 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.797287 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-model-cache\") pod \"b023e403-093c-402a-bb66-752ee5221bfd\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " Jul 07 15:17:01.797487 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.797334 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-home\") pod \"b023e403-093c-402a-bb66-752ee5221bfd\" (UID: \"b023e403-093c-402a-bb66-752ee5221bfd\") " Jul 07 15:17:01.797587 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.797531 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b023e403-093c-402a-bb66-752ee5221bfd" (UID: "b023e403-093c-402a-bb66-752ee5221bfd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:17:01.797587 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.797558 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-model-cache" (OuterVolumeSpecName: "model-cache") pod "b023e403-093c-402a-bb66-752ee5221bfd" (UID: "b023e403-093c-402a-bb66-752ee5221bfd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:17:01.797722 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.797694 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-home" (OuterVolumeSpecName: "home") pod "b023e403-093c-402a-bb66-752ee5221bfd" (UID: "b023e403-093c-402a-bb66-752ee5221bfd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:17:01.797784 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.797707 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-tmp-dir\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:17:01.797784 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.797753 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-model-cache\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:17:01.799189 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.799164 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b023e403-093c-402a-bb66-752ee5221bfd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b023e403-093c-402a-bb66-752ee5221bfd" (UID: "b023e403-093c-402a-bb66-752ee5221bfd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:17:01.799293 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.799266 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b023e403-093c-402a-bb66-752ee5221bfd-kube-api-access-qkqxz" (OuterVolumeSpecName: "kube-api-access-qkqxz") pod "b023e403-093c-402a-bb66-752ee5221bfd" (UID: "b023e403-093c-402a-bb66-752ee5221bfd"). InnerVolumeSpecName "kube-api-access-qkqxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:17:01.799293 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.799276 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-dshm" (OuterVolumeSpecName: "dshm") pod "b023e403-093c-402a-bb66-752ee5221bfd" (UID: "b023e403-093c-402a-bb66-752ee5221bfd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:17:01.898914 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.898857 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b023e403-093c-402a-bb66-752ee5221bfd-tls-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:17:01.898914 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.898908 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkqxz\" (UniqueName: \"kubernetes.io/projected/b023e403-093c-402a-bb66-752ee5221bfd-kube-api-access-qkqxz\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:17:01.898914 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.898922 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-dshm\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:17:01.899168 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:01.898936 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b023e403-093c-402a-bb66-752ee5221bfd-home\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:17:02.034823 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.034788 2583 generic.go:356] "Generic (PLEG): container finished" podID="b023e403-093c-402a-bb66-752ee5221bfd" containerID="6595209af3e570923985ac1f7a48265d3db88376fb542a7dac500ff165101299" exitCode=0 Jul 07 15:17:02.035015 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.034831 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" event={"ID":"b023e403-093c-402a-bb66-752ee5221bfd","Type":"ContainerDied","Data":"6595209af3e570923985ac1f7a48265d3db88376fb542a7dac500ff165101299"} Jul 07 15:17:02.035015 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.034851 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" Jul 07 15:17:02.035015 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.034869 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl" event={"ID":"b023e403-093c-402a-bb66-752ee5221bfd","Type":"ContainerDied","Data":"1f12fbb0993fd086e2b23100b1d46a9be50b89b3c2317f82decf0129f995cf48"} Jul 07 15:17:02.035015 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.034887 2583 scope.go:117] "RemoveContainer" containerID="6595209af3e570923985ac1f7a48265d3db88376fb542a7dac500ff165101299" Jul 07 15:17:02.043841 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.043818 2583 scope.go:117] "RemoveContainer" containerID="6595209af3e570923985ac1f7a48265d3db88376fb542a7dac500ff165101299" Jul 07 15:17:02.044092 ip-10-0-142-144 kubenswrapper[2583]: E0707 15:17:02.044075 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6595209af3e570923985ac1f7a48265d3db88376fb542a7dac500ff165101299\": container with ID starting with 6595209af3e570923985ac1f7a48265d3db88376fb542a7dac500ff165101299 not found: ID does not exist" containerID="6595209af3e570923985ac1f7a48265d3db88376fb542a7dac500ff165101299" Jul 07 15:17:02.044155 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.044104 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6595209af3e570923985ac1f7a48265d3db88376fb542a7dac500ff165101299"} err="failed to get container status \"6595209af3e570923985ac1f7a48265d3db88376fb542a7dac500ff165101299\": rpc error: code = NotFound desc = could not find container \"6595209af3e570923985ac1f7a48265d3db88376fb542a7dac500ff165101299\": container with ID starting with 6595209af3e570923985ac1f7a48265d3db88376fb542a7dac500ff165101299 not found: ID does not exist" Jul 07 15:17:02.057592 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.057531 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl"] Jul 07 15:17:02.070142 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.070114 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-96d90d/sub-test-model-96d90d-kserve-69cf46fd4c-n9snl"] Jul 07 15:17:02.400271 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.400174 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" podUID="530839b6-d1d6-45cd-8235-8e12c7307e74" containerName="main" containerID="cri-o://c58187751f3eb17d710d6c1393572b6ae71a67e40f297fa6897bb18cc4c2dfb4" gracePeriod=45 Jul 07 15:17:02.643888 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.643865 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:17:02.776517 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.776481 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:17:02.776715 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.776542 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:17:02.788613 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.788590 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:17:02.807753 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.807718 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/530839b6-d1d6-45cd-8235-8e12c7307e74-tls-certs\") pod \"530839b6-d1d6-45cd-8235-8e12c7307e74\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " Jul 07 15:17:02.807937 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.807761 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-tmp-dir\") pod \"530839b6-d1d6-45cd-8235-8e12c7307e74\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " Jul 07 15:17:02.807937 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.807807 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-model-cache\") pod \"530839b6-d1d6-45cd-8235-8e12c7307e74\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " Jul 07 15:17:02.807937 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.807830 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-dshm\") pod \"530839b6-d1d6-45cd-8235-8e12c7307e74\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " Jul 07 15:17:02.807937 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.807913 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-home\") pod \"530839b6-d1d6-45cd-8235-8e12c7307e74\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " Jul 07 15:17:02.808178 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.807936 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqgs5\" (UniqueName: \"kubernetes.io/projected/530839b6-d1d6-45cd-8235-8e12c7307e74-kube-api-access-gqgs5\") pod \"530839b6-d1d6-45cd-8235-8e12c7307e74\" (UID: \"530839b6-d1d6-45cd-8235-8e12c7307e74\") " Jul 07 15:17:02.808178 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.808035 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "530839b6-d1d6-45cd-8235-8e12c7307e74" (UID: "530839b6-d1d6-45cd-8235-8e12c7307e74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:17:02.808178 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.808046 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-model-cache" (OuterVolumeSpecName: "model-cache") pod "530839b6-d1d6-45cd-8235-8e12c7307e74" (UID: "530839b6-d1d6-45cd-8235-8e12c7307e74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:17:02.808178 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.808168 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-home" (OuterVolumeSpecName: "home") pod "530839b6-d1d6-45cd-8235-8e12c7307e74" (UID: "530839b6-d1d6-45cd-8235-8e12c7307e74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:17:02.808406 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.808267 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-tmp-dir\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:17:02.808406 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.808295 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-model-cache\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:17:02.808406 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.808303 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-home\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:17:02.809797 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.809772 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/530839b6-d1d6-45cd-8235-8e12c7307e74-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "530839b6-d1d6-45cd-8235-8e12c7307e74" (UID: "530839b6-d1d6-45cd-8235-8e12c7307e74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:17:02.809920 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.809889 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530839b6-d1d6-45cd-8235-8e12c7307e74-kube-api-access-gqgs5" (OuterVolumeSpecName: "kube-api-access-gqgs5") pod "530839b6-d1d6-45cd-8235-8e12c7307e74" (UID: "530839b6-d1d6-45cd-8235-8e12c7307e74"). InnerVolumeSpecName "kube-api-access-gqgs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:17:02.809977 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.809948 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-dshm" (OuterVolumeSpecName: "dshm") pod "530839b6-d1d6-45cd-8235-8e12c7307e74" (UID: "530839b6-d1d6-45cd-8235-8e12c7307e74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:17:02.909603 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.909570 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqgs5\" (UniqueName: \"kubernetes.io/projected/530839b6-d1d6-45cd-8235-8e12c7307e74-kube-api-access-gqgs5\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:17:02.909603 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.909597 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/530839b6-d1d6-45cd-8235-8e12c7307e74-tls-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:17:02.909804 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:02.909680 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/530839b6-d1d6-45cd-8235-8e12c7307e74-dshm\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:17:03.041879 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:03.041796 2583 generic.go:356] "Generic (PLEG): container finished" podID="530839b6-d1d6-45cd-8235-8e12c7307e74" containerID="c58187751f3eb17d710d6c1393572b6ae71a67e40f297fa6897bb18cc4c2dfb4" exitCode=0 Jul 07 15:17:03.041879 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:03.041839 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" Jul 07 15:17:03.042079 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:03.041882 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" event={"ID":"530839b6-d1d6-45cd-8235-8e12c7307e74","Type":"ContainerDied","Data":"c58187751f3eb17d710d6c1393572b6ae71a67e40f297fa6897bb18cc4c2dfb4"} Jul 07 15:17:03.042079 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:03.041926 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf" event={"ID":"530839b6-d1d6-45cd-8235-8e12c7307e74","Type":"ContainerDied","Data":"180b9fb0c750b29eabf455a15bd66a6bd19966569afab014d4b43b3bebc33078"} Jul 07 15:17:03.042079 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:03.041953 2583 scope.go:117] "RemoveContainer" containerID="c58187751f3eb17d710d6c1393572b6ae71a67e40f297fa6897bb18cc4c2dfb4" Jul 07 15:17:03.051874 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:03.051856 2583 scope.go:117] "RemoveContainer" containerID="c58187751f3eb17d710d6c1393572b6ae71a67e40f297fa6897bb18cc4c2dfb4" Jul 07 15:17:03.052120 ip-10-0-142-144 kubenswrapper[2583]: E0707 15:17:03.052104 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c58187751f3eb17d710d6c1393572b6ae71a67e40f297fa6897bb18cc4c2dfb4\": container with ID starting with c58187751f3eb17d710d6c1393572b6ae71a67e40f297fa6897bb18cc4c2dfb4 not found: ID does not exist" containerID="c58187751f3eb17d710d6c1393572b6ae71a67e40f297fa6897bb18cc4c2dfb4" Jul 07 15:17:03.052166 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:03.052128 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c58187751f3eb17d710d6c1393572b6ae71a67e40f297fa6897bb18cc4c2dfb4"} err="failed to get container status \"c58187751f3eb17d710d6c1393572b6ae71a67e40f297fa6897bb18cc4c2dfb4\": rpc error: code = NotFound desc = could not find container \"c58187751f3eb17d710d6c1393572b6ae71a67e40f297fa6897bb18cc4c2dfb4\": container with ID starting with c58187751f3eb17d710d6c1393572b6ae71a67e40f297fa6897bb18cc4c2dfb4 not found: ID does not exist" Jul 07 15:17:03.056300 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:03.056281 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:17:03.064539 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:03.064510 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf"] Jul 07 15:17:03.074243 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:03.074092 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-17f250/sub-test-model-17f250-kserve-598975c465-6z5hf"] Jul 07 15:17:03.792162 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:03.792124 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="530839b6-d1d6-45cd-8235-8e12c7307e74" path="/var/lib/kubelet/pods/530839b6-d1d6-45cd-8235-8e12c7307e74/volumes" Jul 07 15:17:03.792561 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:03.792526 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b023e403-093c-402a-bb66-752ee5221bfd" path="/var/lib/kubelet/pods/b023e403-093c-402a-bb66-752ee5221bfd/volumes" Jul 07 15:17:16.378790 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.378760 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5"] Jul 07 15:17:16.379419 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.379053 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b023e403-093c-402a-bb66-752ee5221bfd" containerName="main" Jul 07 15:17:16.379419 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.379065 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b023e403-093c-402a-bb66-752ee5221bfd" containerName="main" Jul 07 15:17:16.379419 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.379089 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="530839b6-d1d6-45cd-8235-8e12c7307e74" containerName="main" Jul 07 15:17:16.379419 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.379094 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="530839b6-d1d6-45cd-8235-8e12c7307e74" containerName="main" Jul 07 15:17:16.379419 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.379148 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b023e403-093c-402a-bb66-752ee5221bfd" containerName="main" Jul 07 15:17:16.379419 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.379155 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="530839b6-d1d6-45cd-8235-8e12c7307e74" containerName="main" Jul 07 15:17:16.385501 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.385470 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.388864 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.388840 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-17f250\"/\"kube-root-ca.crt\"" Jul 07 15:17:16.389002 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.388861 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-17f250\"/\"openshift-service-ca.crt\"" Jul 07 15:17:16.389002 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.388846 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-17f250\"/\"default-dockercfg-rml5k\"" Jul 07 15:17:16.389002 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.388884 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-17f250\"/\"rate-test-model-17f250-kserve-self-signed-certs\"" Jul 07 15:17:16.395217 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.395196 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5"] Jul 07 15:17:16.422651 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.422629 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-dshm\") pod \"rate-test-model-17f250-kserve-7659bfb54c-nz2l5\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.422787 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.422673 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-model-cache\") pod \"rate-test-model-17f250-kserve-7659bfb54c-nz2l5\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.422787 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.422690 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-tmp-dir\") pod \"rate-test-model-17f250-kserve-7659bfb54c-nz2l5\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.422787 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.422713 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-home\") pod \"rate-test-model-17f250-kserve-7659bfb54c-nz2l5\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.422964 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.422805 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb6e8e8e-125c-4eaa-9115-4d908273c53f-tls-certs\") pod \"rate-test-model-17f250-kserve-7659bfb54c-nz2l5\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.422964 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.422832 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs5pc\" (UniqueName: \"kubernetes.io/projected/bb6e8e8e-125c-4eaa-9115-4d908273c53f-kube-api-access-hs5pc\") pod \"rate-test-model-17f250-kserve-7659bfb54c-nz2l5\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.463709 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.463684 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:17:16.523431 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.523401 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb6e8e8e-125c-4eaa-9115-4d908273c53f-tls-certs\") pod \"rate-test-model-17f250-kserve-7659bfb54c-nz2l5\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.523528 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.523441 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs5pc\" (UniqueName: \"kubernetes.io/projected/bb6e8e8e-125c-4eaa-9115-4d908273c53f-kube-api-access-hs5pc\") pod \"rate-test-model-17f250-kserve-7659bfb54c-nz2l5\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.523528 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.523483 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-dshm\") pod \"rate-test-model-17f250-kserve-7659bfb54c-nz2l5\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.523608 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.523529 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-model-cache\") pod \"rate-test-model-17f250-kserve-7659bfb54c-nz2l5\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.523608 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.523544 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-tmp-dir\") pod \"rate-test-model-17f250-kserve-7659bfb54c-nz2l5\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.523608 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.523572 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-home\") pod \"rate-test-model-17f250-kserve-7659bfb54c-nz2l5\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.523926 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.523900 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-model-cache\") pod \"rate-test-model-17f250-kserve-7659bfb54c-nz2l5\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.524001 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.523952 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-tmp-dir\") pod \"rate-test-model-17f250-kserve-7659bfb54c-nz2l5\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.524043 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.524004 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-home\") pod \"rate-test-model-17f250-kserve-7659bfb54c-nz2l5\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.525649 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.525629 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-dshm\") pod \"rate-test-model-17f250-kserve-7659bfb54c-nz2l5\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.525744 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.525727 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb6e8e8e-125c-4eaa-9115-4d908273c53f-tls-certs\") pod \"rate-test-model-17f250-kserve-7659bfb54c-nz2l5\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.533644 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.533625 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs5pc\" (UniqueName: \"kubernetes.io/projected/bb6e8e8e-125c-4eaa-9115-4d908273c53f-kube-api-access-hs5pc\") pod \"rate-test-model-17f250-kserve-7659bfb54c-nz2l5\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.695815 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.695740 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:16.824736 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:16.824712 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5"] Jul 07 15:17:16.826037 ip-10-0-142-144 kubenswrapper[2583]: W0707 15:17:16.826010 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb6e8e8e_125c_4eaa_9115_4d908273c53f.slice/crio-af27128e8cfe2689cbf471efa370e759b4325ffbb2520a60311bb5403a1bb06a WatchSource:0}: Error finding container af27128e8cfe2689cbf471efa370e759b4325ffbb2520a60311bb5403a1bb06a: Status 404 returned error can't find the container with id af27128e8cfe2689cbf471efa370e759b4325ffbb2520a60311bb5403a1bb06a Jul 07 15:17:17.112739 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:17.112701 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" event={"ID":"bb6e8e8e-125c-4eaa-9115-4d908273c53f","Type":"ContainerStarted","Data":"4c364f6336622c37c20838dcbe409ae70126c5e7db9ad4b099edb99d3d6318f1"} Jul 07 15:17:17.112895 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:17.112750 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" event={"ID":"bb6e8e8e-125c-4eaa-9115-4d908273c53f","Type":"ContainerStarted","Data":"af27128e8cfe2689cbf471efa370e759b4325ffbb2520a60311bb5403a1bb06a"} Jul 07 15:17:17.139512 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:17.139470 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" podStartSLOduration=1.139454993 podStartE2EDuration="1.139454993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:17:17.137743569 +0000 UTC m=+2570.048937247" watchObservedRunningTime="2026-07-07 15:17:17.139454993 +0000 UTC m=+2570.050648670" Jul 07 15:17:26.696622 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:26.696580 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:26.697021 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:26.696634 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:26.709272 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:26.709246 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:27.173376 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:27.173333 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:17:30.734079 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:30.734025 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:17:52.656950 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:52.656918 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:17:52.757314 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:17:52.757291 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:18:08.324508 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:08.324477 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:18:08.424095 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:08.424063 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:18:11.538498 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:11.538468 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r"] Jul 07 15:18:11.561376 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:11.561315 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:18:12.545984 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:12.545943 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5"] Jul 07 15:18:12.564075 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:12.564050 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:18:14.474027 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:14.473991 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:18:14.543963 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:14.543935 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:18:17.298044 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:17.298013 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:18:17.398586 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:17.398554 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:18:19.096867 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:19.096807 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:18:26.558018 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.557956 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" podUID="23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc" containerName="main" containerID="cri-o://6ff66d4bc4e82674c741be15967f8e272a724a317c44c7ca57439f886038a4a6" gracePeriod=45 Jul 07 15:18:26.807764 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.807733 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:18:26.887480 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.887416 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-tls-certs\") pod \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " Jul 07 15:18:26.887480 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.887463 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4kxp\" (UniqueName: \"kubernetes.io/projected/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-kube-api-access-q4kxp\") pod \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " Jul 07 15:18:26.887653 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.887634 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-home\") pod \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " Jul 07 15:18:26.887705 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.887670 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-tmp-dir\") pod \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " Jul 07 15:18:26.887705 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.887689 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-dshm\") pod \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " Jul 07 15:18:26.887823 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.887714 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-model-cache\") pod \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\" (UID: \"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc\") " Jul 07 15:18:26.887965 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.887928 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc" (UID: "23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:18:26.887965 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.887948 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-home" (OuterVolumeSpecName: "home") pod "23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc" (UID: "23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:18:26.888106 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.888084 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc" (UID: "23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:18:26.888160 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.888131 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-home\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:18:26.888160 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.888151 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-tmp-dir\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:18:26.889793 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.889774 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-dshm" (OuterVolumeSpecName: "dshm") pod "23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc" (UID: "23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:18:26.890018 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.890000 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc" (UID: "23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:18:26.890095 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.890019 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-kube-api-access-q4kxp" (OuterVolumeSpecName: "kube-api-access-q4kxp") pod "23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc" (UID: "23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc"). InnerVolumeSpecName "kube-api-access-q4kxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:18:26.988788 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.988767 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-dshm\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:18:26.988788 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.988789 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-model-cache\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:18:26.988916 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.988799 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-tls-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:18:26.988916 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:26.988808 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4kxp\" (UniqueName: \"kubernetes.io/projected/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc-kube-api-access-q4kxp\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:18:27.458689 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.458657 2583 generic.go:356] "Generic (PLEG): container finished" podID="23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc" containerID="6ff66d4bc4e82674c741be15967f8e272a724a317c44c7ca57439f886038a4a6" exitCode=0 Jul 07 15:18:27.458870 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.458695 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" Jul 07 15:18:27.458870 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.458724 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" event={"ID":"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc","Type":"ContainerDied","Data":"6ff66d4bc4e82674c741be15967f8e272a724a317c44c7ca57439f886038a4a6"} Jul 07 15:18:27.458870 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.458765 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r" event={"ID":"23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc","Type":"ContainerDied","Data":"653e1bec6587f0d43138f432f5ae89c57235f490ae15b94055db64cdecb4b049"} Jul 07 15:18:27.458870 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.458786 2583 scope.go:117] "RemoveContainer" containerID="6ff66d4bc4e82674c741be15967f8e272a724a317c44c7ca57439f886038a4a6" Jul 07 15:18:27.467549 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.467537 2583 scope.go:117] "RemoveContainer" containerID="6ff66d4bc4e82674c741be15967f8e272a724a317c44c7ca57439f886038a4a6" Jul 07 15:18:27.467833 ip-10-0-142-144 kubenswrapper[2583]: E0707 15:18:27.467812 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff66d4bc4e82674c741be15967f8e272a724a317c44c7ca57439f886038a4a6\": container with ID starting with 6ff66d4bc4e82674c741be15967f8e272a724a317c44c7ca57439f886038a4a6 not found: ID does not exist" containerID="6ff66d4bc4e82674c741be15967f8e272a724a317c44c7ca57439f886038a4a6" Jul 07 15:18:27.467929 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.467838 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff66d4bc4e82674c741be15967f8e272a724a317c44c7ca57439f886038a4a6"} err="failed to get container status \"6ff66d4bc4e82674c741be15967f8e272a724a317c44c7ca57439f886038a4a6\": rpc error: code = NotFound desc = could not find container \"6ff66d4bc4e82674c741be15967f8e272a724a317c44c7ca57439f886038a4a6\": container with ID starting with 6ff66d4bc4e82674c741be15967f8e272a724a317c44c7ca57439f886038a4a6 not found: ID does not exist" Jul 07 15:18:27.479952 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.479927 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r"] Jul 07 15:18:27.486350 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.486332 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-96d90d/rate-test-model-96d90d-kserve-7c4c66d8b7-g5z5r"] Jul 07 15:18:27.565794 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.565769 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" podUID="bb6e8e8e-125c-4eaa-9115-4d908273c53f" containerName="main" containerID="cri-o://4c364f6336622c37c20838dcbe409ae70126c5e7db9ad4b099edb99d3d6318f1" gracePeriod=45 Jul 07 15:18:27.793073 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.793041 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc" path="/var/lib/kubelet/pods/23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc/volumes" Jul 07 15:18:27.813247 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.813224 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:18:27.895371 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.895328 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-tmp-dir\") pod \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " Jul 07 15:18:27.895508 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.895394 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb6e8e8e-125c-4eaa-9115-4d908273c53f-tls-certs\") pod \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " Jul 07 15:18:27.895508 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.895430 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs5pc\" (UniqueName: \"kubernetes.io/projected/bb6e8e8e-125c-4eaa-9115-4d908273c53f-kube-api-access-hs5pc\") pod \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " Jul 07 15:18:27.895508 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.895484 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-model-cache\") pod \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " Jul 07 15:18:27.895642 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.895582 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-dshm\") pod \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " Jul 07 15:18:27.895642 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.895611 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-home\") pod \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\" (UID: \"bb6e8e8e-125c-4eaa-9115-4d908273c53f\") " Jul 07 15:18:27.895642 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.895616 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb6e8e8e-125c-4eaa-9115-4d908273c53f" (UID: "bb6e8e8e-125c-4eaa-9115-4d908273c53f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:18:27.895813 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.895747 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb6e8e8e-125c-4eaa-9115-4d908273c53f" (UID: "bb6e8e8e-125c-4eaa-9115-4d908273c53f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:18:27.895877 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.895821 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-home" (OuterVolumeSpecName: "home") pod "bb6e8e8e-125c-4eaa-9115-4d908273c53f" (UID: "bb6e8e8e-125c-4eaa-9115-4d908273c53f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:18:27.895933 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.895916 2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-model-cache\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:18:27.895933 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.895928 2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-home\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:18:27.896018 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.895943 2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-tmp-dir\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:18:27.897533 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.897504 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-dshm" (OuterVolumeSpecName: "dshm") pod "bb6e8e8e-125c-4eaa-9115-4d908273c53f" (UID: "bb6e8e8e-125c-4eaa-9115-4d908273c53f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:18:27.897636 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.897531 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb6e8e8e-125c-4eaa-9115-4d908273c53f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb6e8e8e-125c-4eaa-9115-4d908273c53f" (UID: "bb6e8e8e-125c-4eaa-9115-4d908273c53f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:18:27.897636 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.897561 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6e8e8e-125c-4eaa-9115-4d908273c53f-kube-api-access-hs5pc" (OuterVolumeSpecName: "kube-api-access-hs5pc") pod "bb6e8e8e-125c-4eaa-9115-4d908273c53f" (UID: "bb6e8e8e-125c-4eaa-9115-4d908273c53f"). InnerVolumeSpecName "kube-api-access-hs5pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:18:27.996964 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.996893 2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb6e8e8e-125c-4eaa-9115-4d908273c53f-dshm\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:18:27.996964 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.996917 2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb6e8e8e-125c-4eaa-9115-4d908273c53f-tls-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:18:27.996964 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:27.996927 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs5pc\" (UniqueName: \"kubernetes.io/projected/bb6e8e8e-125c-4eaa-9115-4d908273c53f-kube-api-access-hs5pc\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:18:28.463899 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:28.463811 2583 generic.go:356] "Generic (PLEG): container finished" podID="bb6e8e8e-125c-4eaa-9115-4d908273c53f" containerID="4c364f6336622c37c20838dcbe409ae70126c5e7db9ad4b099edb99d3d6318f1" exitCode=0 Jul 07 15:18:28.463899 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:28.463861 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" Jul 07 15:18:28.464100 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:28.463906 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" event={"ID":"bb6e8e8e-125c-4eaa-9115-4d908273c53f","Type":"ContainerDied","Data":"4c364f6336622c37c20838dcbe409ae70126c5e7db9ad4b099edb99d3d6318f1"} Jul 07 15:18:28.464100 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:28.463943 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5" event={"ID":"bb6e8e8e-125c-4eaa-9115-4d908273c53f","Type":"ContainerDied","Data":"af27128e8cfe2689cbf471efa370e759b4325ffbb2520a60311bb5403a1bb06a"} Jul 07 15:18:28.464100 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:28.463959 2583 scope.go:117] "RemoveContainer" containerID="4c364f6336622c37c20838dcbe409ae70126c5e7db9ad4b099edb99d3d6318f1" Jul 07 15:18:28.472889 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:28.472872 2583 scope.go:117] "RemoveContainer" containerID="4c364f6336622c37c20838dcbe409ae70126c5e7db9ad4b099edb99d3d6318f1" Jul 07 15:18:28.473147 ip-10-0-142-144 kubenswrapper[2583]: E0707 15:18:28.473129 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c364f6336622c37c20838dcbe409ae70126c5e7db9ad4b099edb99d3d6318f1\": container with ID starting with 4c364f6336622c37c20838dcbe409ae70126c5e7db9ad4b099edb99d3d6318f1 not found: ID does not exist" containerID="4c364f6336622c37c20838dcbe409ae70126c5e7db9ad4b099edb99d3d6318f1" Jul 07 15:18:28.473193 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:28.473153 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c364f6336622c37c20838dcbe409ae70126c5e7db9ad4b099edb99d3d6318f1"} err="failed to get container status \"4c364f6336622c37c20838dcbe409ae70126c5e7db9ad4b099edb99d3d6318f1\": rpc error: code = NotFound desc = could not find container \"4c364f6336622c37c20838dcbe409ae70126c5e7db9ad4b099edb99d3d6318f1\": container with ID starting with 4c364f6336622c37c20838dcbe409ae70126c5e7db9ad4b099edb99d3d6318f1 not found: ID does not exist" Jul 07 15:18:28.484569 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:28.484547 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5"] Jul 07 15:18:28.489426 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:28.489408 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-17f250/rate-test-model-17f250-kserve-7659bfb54c-nz2l5"] Jul 07 15:18:29.793009 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:29.792966 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb6e8e8e-125c-4eaa-9115-4d908273c53f" path="/var/lib/kubelet/pods/bb6e8e8e-125c-4eaa-9115-4d908273c53f/volumes" Jul 07 15:18:56.039511 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:56.039477 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:18:56.313076 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:56.312997 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc"] Jul 07 15:18:56.313247 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:56.313221 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" podUID="d7e2783d-2112-4569-a0c8-674931cc3386" containerName="istio-proxy" containerID="cri-o://8a5a41ada92a41673e99cae80eb2da7f045d17495b64a9ed2469f74e2b11ea7b" gracePeriod=30 Jul 07 15:18:56.378486 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:56.378459 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:18:57.604231 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:57.604194 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:18:57.672089 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:18:57.672062 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:19:01.561671 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.561648 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:19:01.632131 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.632046 2583 generic.go:356] "Generic (PLEG): container finished" podID="d7e2783d-2112-4569-a0c8-674931cc3386" containerID="8a5a41ada92a41673e99cae80eb2da7f045d17495b64a9ed2469f74e2b11ea7b" exitCode=0 Jul 07 15:19:01.632131 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.632102 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" Jul 07 15:19:01.632327 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.632133 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" event={"ID":"d7e2783d-2112-4569-a0c8-674931cc3386","Type":"ContainerDied","Data":"8a5a41ada92a41673e99cae80eb2da7f045d17495b64a9ed2469f74e2b11ea7b"} Jul 07 15:19:01.632327 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.632180 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc" event={"ID":"d7e2783d-2112-4569-a0c8-674931cc3386","Type":"ContainerDied","Data":"91f52b2e2dee0d749516bcc1e68d5f82bdec03506476744160a18a3fab69397b"} Jul 07 15:19:01.632327 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.632206 2583 scope.go:117] "RemoveContainer" containerID="8a5a41ada92a41673e99cae80eb2da7f045d17495b64a9ed2469f74e2b11ea7b" Jul 07 15:19:01.640493 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.640474 2583 scope.go:117] "RemoveContainer" containerID="f8b4c2799a829f9e27da74ce550e51e88fd8d6740b5141847aeb796ab1ce8adf" Jul 07 15:19:01.647855 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.647839 2583 scope.go:117] "RemoveContainer" containerID="8a5a41ada92a41673e99cae80eb2da7f045d17495b64a9ed2469f74e2b11ea7b" Jul 07 15:19:01.648096 ip-10-0-142-144 kubenswrapper[2583]: E0707 15:19:01.648075 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5a41ada92a41673e99cae80eb2da7f045d17495b64a9ed2469f74e2b11ea7b\": container with ID starting with 8a5a41ada92a41673e99cae80eb2da7f045d17495b64a9ed2469f74e2b11ea7b not found: ID does not exist" containerID="8a5a41ada92a41673e99cae80eb2da7f045d17495b64a9ed2469f74e2b11ea7b" Jul 07 15:19:01.648156 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.648106 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5a41ada92a41673e99cae80eb2da7f045d17495b64a9ed2469f74e2b11ea7b"} err="failed to get container status \"8a5a41ada92a41673e99cae80eb2da7f045d17495b64a9ed2469f74e2b11ea7b\": rpc error: code = NotFound desc = could not find container \"8a5a41ada92a41673e99cae80eb2da7f045d17495b64a9ed2469f74e2b11ea7b\": container with ID starting with 8a5a41ada92a41673e99cae80eb2da7f045d17495b64a9ed2469f74e2b11ea7b not found: ID does not exist" Jul 07 15:19:01.648156 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.648123 2583 scope.go:117] "RemoveContainer" containerID="f8b4c2799a829f9e27da74ce550e51e88fd8d6740b5141847aeb796ab1ce8adf" Jul 07 15:19:01.648314 ip-10-0-142-144 kubenswrapper[2583]: E0707 15:19:01.648296 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b4c2799a829f9e27da74ce550e51e88fd8d6740b5141847aeb796ab1ce8adf\": container with ID starting with f8b4c2799a829f9e27da74ce550e51e88fd8d6740b5141847aeb796ab1ce8adf not found: ID does not exist" containerID="f8b4c2799a829f9e27da74ce550e51e88fd8d6740b5141847aeb796ab1ce8adf" Jul 07 15:19:01.648391 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.648321 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b4c2799a829f9e27da74ce550e51e88fd8d6740b5141847aeb796ab1ce8adf"} err="failed to get container status \"f8b4c2799a829f9e27da74ce550e51e88fd8d6740b5141847aeb796ab1ce8adf\": rpc error: code = NotFound desc = could not find container \"f8b4c2799a829f9e27da74ce550e51e88fd8d6740b5141847aeb796ab1ce8adf\": container with ID starting with f8b4c2799a829f9e27da74ce550e51e88fd8d6740b5141847aeb796ab1ce8adf not found: ID does not exist" Jul 07 15:19:01.653550 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.653535 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-credential-socket\") pod \"d7e2783d-2112-4569-a0c8-674931cc3386\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " Jul 07 15:19:01.653603 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.653574 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e2783d-2112-4569-a0c8-674931cc3386-istio-token\") pod \"d7e2783d-2112-4569-a0c8-674931cc3386\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " Jul 07 15:19:01.653637 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.653601 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxcst\" (UniqueName: \"kubernetes.io/projected/d7e2783d-2112-4569-a0c8-674931cc3386-kube-api-access-zxcst\") pod \"d7e2783d-2112-4569-a0c8-674931cc3386\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " Jul 07 15:19:01.653737 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.653719 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-workload-certs\") pod \"d7e2783d-2112-4569-a0c8-674931cc3386\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " Jul 07 15:19:01.653794 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.653756 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e2783d-2112-4569-a0c8-674931cc3386-istio-podinfo\") pod \"d7e2783d-2112-4569-a0c8-674931cc3386\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " Jul 07 15:19:01.653794 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.653769 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d7e2783d-2112-4569-a0c8-674931cc3386" (UID: "d7e2783d-2112-4569-a0c8-674931cc3386"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:19:01.653895 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.653797 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-istio-data\") pod \"d7e2783d-2112-4569-a0c8-674931cc3386\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " Jul 07 15:19:01.653895 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.653822 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e2783d-2112-4569-a0c8-674931cc3386-istiod-ca-cert\") pod \"d7e2783d-2112-4569-a0c8-674931cc3386\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " Jul 07 15:19:01.653895 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.653850 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-workload-socket\") pod \"d7e2783d-2112-4569-a0c8-674931cc3386\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " Jul 07 15:19:01.654037 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.653909 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-istio-envoy\") pod \"d7e2783d-2112-4569-a0c8-674931cc3386\" (UID: \"d7e2783d-2112-4569-a0c8-674931cc3386\") " Jul 07 15:19:01.654037 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.653983 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d7e2783d-2112-4569-a0c8-674931cc3386" (UID: "d7e2783d-2112-4569-a0c8-674931cc3386"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:19:01.654140 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.654114 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d7e2783d-2112-4569-a0c8-674931cc3386" (UID: "d7e2783d-2112-4569-a0c8-674931cc3386"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:19:01.654198 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.654160 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e2783d-2112-4569-a0c8-674931cc3386-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d7e2783d-2112-4569-a0c8-674931cc3386" (UID: "d7e2783d-2112-4569-a0c8-674931cc3386"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:19:01.654257 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.654228 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-workload-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:19:01.654257 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.654242 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-workload-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:19:01.654257 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.654251 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-credential-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:19:01.654848 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.654822 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-istio-data" (OuterVolumeSpecName: "istio-data") pod "d7e2783d-2112-4569-a0c8-674931cc3386" (UID: "d7e2783d-2112-4569-a0c8-674931cc3386"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:19:01.655640 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.655615 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e2783d-2112-4569-a0c8-674931cc3386-kube-api-access-zxcst" (OuterVolumeSpecName: "kube-api-access-zxcst") pod "d7e2783d-2112-4569-a0c8-674931cc3386" (UID: "d7e2783d-2112-4569-a0c8-674931cc3386"). InnerVolumeSpecName "kube-api-access-zxcst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:19:01.655932 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.655906 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7e2783d-2112-4569-a0c8-674931cc3386-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d7e2783d-2112-4569-a0c8-674931cc3386" (UID: "d7e2783d-2112-4569-a0c8-674931cc3386"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 15:19:01.656144 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.656129 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e2783d-2112-4569-a0c8-674931cc3386-istio-token" (OuterVolumeSpecName: "istio-token") pod "d7e2783d-2112-4569-a0c8-674931cc3386" (UID: "d7e2783d-2112-4569-a0c8-674931cc3386"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:19:01.656144 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.656135 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d7e2783d-2112-4569-a0c8-674931cc3386" (UID: "d7e2783d-2112-4569-a0c8-674931cc3386"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:19:01.755117 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.755089 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e2783d-2112-4569-a0c8-674931cc3386-istio-token\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:19:01.755117 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.755112 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxcst\" (UniqueName: \"kubernetes.io/projected/d7e2783d-2112-4569-a0c8-674931cc3386-kube-api-access-zxcst\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:19:01.755117 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.755121 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e2783d-2112-4569-a0c8-674931cc3386-istio-podinfo\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:19:01.755308 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.755129 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-istio-data\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:19:01.755308 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.755138 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e2783d-2112-4569-a0c8-674931cc3386-istiod-ca-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:19:01.755308 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.755145 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e2783d-2112-4569-a0c8-674931cc3386-istio-envoy\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Jul 07 15:19:01.951272 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.951203 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc"] Jul 07 15:19:01.955014 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:01.954992 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-96d90d-openshift-default-7b947b59ff-whhzc"] Jul 07 15:19:03.196624 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:03.196592 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:19:03.296876 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:03.296842 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:19:03.465835 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:03.465757 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:19:03.791327 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:03.791290 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7e2783d-2112-4569-a0c8-674931cc3386" path="/var/lib/kubelet/pods/d7e2783d-2112-4569-a0c8-674931cc3386/volumes" Jul 07 15:19:04.101629 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:04.101550 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f5b44b88f-w7cpd_497257b3-dfd7-4c1e-9453-a86f4a626df5/authorino/0.log" Jul 07 15:19:04.926558 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:04.926527 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:19:09.099537 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:09.099508 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29723940-rd9nj_835568d6-74c8-48f0-8e21-e4550d1a45d1/cleanup/0.log" Jul 07 15:19:09.455737 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:09.455667 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-9t5l7_7a82903d-7b22-4a68-bc89-4d2b06b7d5d4/server/0.log" Jul 07 15:19:09.573349 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:09.573323 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-87x9g_5a37d477-1a1c-4084-8b6b-735c1e32a091/manager/0.log" Jul 07 15:19:09.948550 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:09.948522 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-655bb48ff6-rjktz_239c61dc-8cbd-4bca-8407-0f25210678b2/copy-manifests/0.log" Jul 07 15:19:09.955391 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:09.955370 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-655bb48ff6-rjktz_239c61dc-8cbd-4bca-8407-0f25210678b2/manager/0.log" Jul 07 15:19:10.860390 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:10.860348 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn_ee8d0df4-fa26-41ff-ab72-e891764fdd79/util/0.log" Jul 07 15:19:10.867175 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:10.867150 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn_ee8d0df4-fa26-41ff-ab72-e891764fdd79/pull/0.log" Jul 07 15:19:10.873827 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:10.873811 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn_ee8d0df4-fa26-41ff-ab72-e891764fdd79/extract/0.log" Jul 07 15:19:10.992099 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:10.992076 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h_7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c/util/0.log" Jul 07 15:19:10.999342 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:10.999322 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h_7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c/pull/0.log" Jul 07 15:19:11.005831 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:11.005811 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h_7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c/extract/0.log" Jul 07 15:19:11.124973 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:11.124909 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh_3d209042-711f-4e4a-9ccc-c1c3d4963546/util/0.log" Jul 07 15:19:11.130723 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:11.130704 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh_3d209042-711f-4e4a-9ccc-c1c3d4963546/pull/0.log" Jul 07 15:19:11.136128 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:11.136114 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh_3d209042-711f-4e4a-9ccc-c1c3d4963546/extract/0.log" Jul 07 15:19:11.254901 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:11.254875 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr_d345aba4-566d-4088-a96a-953b6983c9e7/util/0.log" Jul 07 15:19:11.261054 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:11.261036 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr_d345aba4-566d-4088-a96a-953b6983c9e7/pull/0.log" Jul 07 15:19:11.267049 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:11.267033 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr_d345aba4-566d-4088-a96a-953b6983c9e7/extract/0.log" Jul 07 15:19:11.389982 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:11.389903 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f5b44b88f-w7cpd_497257b3-dfd7-4c1e-9453-a86f4a626df5/authorino/0.log" Jul 07 15:19:13.004703 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:13.004676 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f6b8bcfd-vpbmb_e28b42f3-a1c2-4fa1-9479-b20be56f35c3/kube-auth-proxy/0.log" Jul 07 15:19:13.128259 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:13.128229 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-rsgc7_ffb40f68-9d64-4f10-ad88-68cfb5ec23ba/istio-proxy/1.log" Jul 07 15:19:14.173407 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:14.173378 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db2h4rh_08121fd0-34c3-4e39-88ba-c64fda4ce303/main/0.log" Jul 07 15:19:14.404060 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:14.404033 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-rcnnn_5069baf9-98db-4e4b-a0f8-1a342f61d1f4/main/0.log" Jul 07 15:19:18.678935 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.678900 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-psm49/must-gather-bbr58"] Jul 07 15:19:18.679311 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.679172 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7e2783d-2112-4569-a0c8-674931cc3386" containerName="istio-proxy" Jul 07 15:19:18.679311 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.679183 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7e2783d-2112-4569-a0c8-674931cc3386" containerName="istio-proxy" Jul 07 15:19:18.679311 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.679196 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc" containerName="main" Jul 07 15:19:18.679311 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.679201 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc" containerName="main" Jul 07 15:19:18.679311 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.679214 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb6e8e8e-125c-4eaa-9115-4d908273c53f" containerName="main" Jul 07 15:19:18.679311 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.679220 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb6e8e8e-125c-4eaa-9115-4d908273c53f" containerName="main" Jul 07 15:19:18.679311 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.679269 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7e2783d-2112-4569-a0c8-674931cc3386" containerName="istio-proxy" Jul 07 15:19:18.679311 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.679279 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="23425ba4-3ce5-49fb-a6c4-d2b7d80e90cc" containerName="main" Jul 07 15:19:18.679311 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.679286 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb6e8e8e-125c-4eaa-9115-4d908273c53f" containerName="main" Jul 07 15:19:18.679643 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.679344 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7e2783d-2112-4569-a0c8-674931cc3386" containerName="istio-proxy" Jul 07 15:19:18.679643 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.679349 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7e2783d-2112-4569-a0c8-674931cc3386" containerName="istio-proxy" Jul 07 15:19:18.679643 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.679422 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7e2783d-2112-4569-a0c8-674931cc3386" containerName="istio-proxy" Jul 07 15:19:18.682312 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.682293 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-psm49/must-gather-bbr58" Jul 07 15:19:18.684738 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.684716 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-psm49\"/\"default-dockercfg-rg4cq\"" Jul 07 15:19:18.685455 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.685436 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-psm49\"/\"kube-root-ca.crt\"" Jul 07 15:19:18.685552 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.685456 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-psm49\"/\"openshift-service-ca.crt\"" Jul 07 15:19:18.691222 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.691198 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-psm49/must-gather-bbr58"] Jul 07 15:19:18.700667 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.700646 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:19:18.771231 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.771209 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" limit=1024 Jul 07 15:19:18.785032 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.785008 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcsht\" (UniqueName: \"kubernetes.io/projected/498931c3-374d-47a1-80a0-a2beb08c9828-kube-api-access-lcsht\") pod \"must-gather-bbr58\" (UID: \"498931c3-374d-47a1-80a0-a2beb08c9828\") " pod="openshift-must-gather-psm49/must-gather-bbr58" Jul 07 15:19:18.785120 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.785071 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/498931c3-374d-47a1-80a0-a2beb08c9828-must-gather-output\") pod \"must-gather-bbr58\" (UID: \"498931c3-374d-47a1-80a0-a2beb08c9828\") " pod="openshift-must-gather-psm49/must-gather-bbr58" Jul 07 15:19:18.886246 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.886223 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/498931c3-374d-47a1-80a0-a2beb08c9828-must-gather-output\") pod \"must-gather-bbr58\" (UID: \"498931c3-374d-47a1-80a0-a2beb08c9828\") " pod="openshift-must-gather-psm49/must-gather-bbr58" Jul 07 15:19:18.886324 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.886258 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcsht\" (UniqueName: \"kubernetes.io/projected/498931c3-374d-47a1-80a0-a2beb08c9828-kube-api-access-lcsht\") pod \"must-gather-bbr58\" (UID: \"498931c3-374d-47a1-80a0-a2beb08c9828\") " pod="openshift-must-gather-psm49/must-gather-bbr58" Jul 07 15:19:18.886602 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.886586 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/498931c3-374d-47a1-80a0-a2beb08c9828-must-gather-output\") pod \"must-gather-bbr58\" (UID: \"498931c3-374d-47a1-80a0-a2beb08c9828\") " pod="openshift-must-gather-psm49/must-gather-bbr58" Jul 07 15:19:18.898419 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.898391 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcsht\" (UniqueName: \"kubernetes.io/projected/498931c3-374d-47a1-80a0-a2beb08c9828-kube-api-access-lcsht\") pod \"must-gather-bbr58\" (UID: \"498931c3-374d-47a1-80a0-a2beb08c9828\") " pod="openshift-must-gather-psm49/must-gather-bbr58" Jul 07 15:19:18.991892 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:18.991875 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-psm49/must-gather-bbr58" Jul 07 15:19:19.114979 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:19.114954 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-psm49/must-gather-bbr58"] Jul 07 15:19:19.116784 ip-10-0-142-144 kubenswrapper[2583]: W0707 15:19:19.116756 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod498931c3_374d_47a1_80a0_a2beb08c9828.slice/crio-56a1b41431f2be6625c0f43580163fd195c0e27a188e1e77811f228b8c85ac5a WatchSource:0}: Error finding container 56a1b41431f2be6625c0f43580163fd195c0e27a188e1e77811f228b8c85ac5a: Status 404 returned error can't find the container with id 56a1b41431f2be6625c0f43580163fd195c0e27a188e1e77811f228b8c85ac5a Jul 07 15:19:19.118551 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:19.118533 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 15:19:19.716010 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:19.715970 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-psm49/must-gather-bbr58" event={"ID":"498931c3-374d-47a1-80a0-a2beb08c9828","Type":"ContainerStarted","Data":"56a1b41431f2be6625c0f43580163fd195c0e27a188e1e77811f228b8c85ac5a"} Jul 07 15:19:20.724248 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:20.724211 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-psm49/must-gather-bbr58" event={"ID":"498931c3-374d-47a1-80a0-a2beb08c9828","Type":"ContainerStarted","Data":"127f255c5103650bc5bb8b482afe2069dc79e830f3106729ad19bee1e87b7ca0"} Jul 07 15:19:20.724724 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:20.724257 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-psm49/must-gather-bbr58" event={"ID":"498931c3-374d-47a1-80a0-a2beb08c9828","Type":"ContainerStarted","Data":"d406427e764ab8d0952e0547e299d79e8cf81ad0a2de60355d1d0445eefb7f1f"} Jul 07 15:19:20.740572 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:20.740514 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-psm49/must-gather-bbr58" podStartSLOduration=1.7687402460000001 podStartE2EDuration="2.740494048s" totalImagesPullingTime="971.753802ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:19:20.739118757 +0000 UTC m=+2693.650312446" watchObservedRunningTime="2026-07-07 15:19:20.740494048 +0000 UTC m=+2693.651687724" Jul 07 15:19:21.721814 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:21.721778 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sjd5c_3a0f179d-66e3-41bd-aa10-71efe8001185/global-pull-secret-syncer/0.log" Jul 07 15:19:21.774196 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:21.774164 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2rp28_2ded5607-54b2-4314-bb3f-443bed8150e6/konnectivity-agent/0.log" Jul 07 15:19:21.906042 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:21.906009 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-144.ec2.internal_2f554e2b8de4942207b69bb5df078360/haproxy/0.log" Jul 07 15:19:26.074284 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:26.074216 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn_ee8d0df4-fa26-41ff-ab72-e891764fdd79/extract/0.log" Jul 07 15:19:26.108902 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:26.108868 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn_ee8d0df4-fa26-41ff-ab72-e891764fdd79/util/0.log" Jul 07 15:19:26.133413 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:26.133334 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978tcn_ee8d0df4-fa26-41ff-ab72-e891764fdd79/pull/0.log" Jul 07 15:19:26.160926 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:26.160889 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h_7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c/extract/0.log" Jul 07 15:19:26.187234 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:26.187155 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h_7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c/util/0.log" Jul 07 15:19:26.213331 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:26.213297 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wvk9h_7f9751ba-a2f8-4c7b-b3f9-28ef1ce1280c/pull/0.log" Jul 07 15:19:26.243354 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:26.243315 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh_3d209042-711f-4e4a-9ccc-c1c3d4963546/extract/0.log" Jul 07 15:19:26.275978 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:26.275945 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh_3d209042-711f-4e4a-9ccc-c1c3d4963546/util/0.log" Jul 07 15:19:26.300170 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:26.300105 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gswhh_3d209042-711f-4e4a-9ccc-c1c3d4963546/pull/0.log" Jul 07 15:19:26.329460 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:26.329379 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr_d345aba4-566d-4088-a96a-953b6983c9e7/extract/0.log" Jul 07 15:19:26.352864 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:26.352839 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr_d345aba4-566d-4088-a96a-953b6983c9e7/util/0.log" Jul 07 15:19:26.376589 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:26.376556 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lq4wr_d345aba4-566d-4088-a96a-953b6983c9e7/pull/0.log" Jul 07 15:19:26.750383 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:26.749842 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f5b44b88f-w7cpd_497257b3-dfd7-4c1e-9453-a86f4a626df5/authorino/0.log" Jul 07 15:19:27.818382 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:27.815241 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 15:19:27.818974 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:27.818709 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 15:19:29.350133 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:29.350100 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n7m8n_93c7f772-cc01-44d3-8ff2-f4c2bfedb630/node-exporter/0.log" Jul 07 15:19:29.373723 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:29.373697 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n7m8n_93c7f772-cc01-44d3-8ff2-f4c2bfedb630/kube-rbac-proxy/0.log" Jul 07 15:19:29.398609 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:29.398570 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n7m8n_93c7f772-cc01-44d3-8ff2-f4c2bfedb630/init-textfile/0.log" Jul 07 15:19:29.522190 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:29.522160 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-kjndn_b6a35f32-7606-4895-af52-5345da42bf07/kube-rbac-proxy-main/0.log" Jul 07 15:19:29.545641 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:29.545612 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-kjndn_b6a35f32-7606-4895-af52-5345da42bf07/kube-rbac-proxy-self/0.log" Jul 07 15:19:29.568806 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:29.568768 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-kjndn_b6a35f32-7606-4895-af52-5345da42bf07/openshift-state-metrics/0.log" Jul 07 15:19:31.410575 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.410508 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v"] Jul 07 15:19:31.416839 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.416810 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" Jul 07 15:19:31.429528 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.429503 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v"] Jul 07 15:19:31.517830 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.517787 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8d6563b-9138-41e4-ab7c-889723489e42-sys\") pod \"perf-node-gather-daemonset-62p5v\" (UID: \"d8d6563b-9138-41e4-ab7c-889723489e42\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" Jul 07 15:19:31.518136 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.518109 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8d6563b-9138-41e4-ab7c-889723489e42-podres\") pod \"perf-node-gather-daemonset-62p5v\" (UID: \"d8d6563b-9138-41e4-ab7c-889723489e42\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" Jul 07 15:19:31.518307 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.518288 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8d6563b-9138-41e4-ab7c-889723489e42-lib-modules\") pod \"perf-node-gather-daemonset-62p5v\" (UID: \"d8d6563b-9138-41e4-ab7c-889723489e42\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" Jul 07 15:19:31.518465 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.518448 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfwvw\" (UniqueName: \"kubernetes.io/projected/d8d6563b-9138-41e4-ab7c-889723489e42-kube-api-access-rfwvw\") pod \"perf-node-gather-daemonset-62p5v\" (UID: \"d8d6563b-9138-41e4-ab7c-889723489e42\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" Jul 07 15:19:31.518623 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.518606 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8d6563b-9138-41e4-ab7c-889723489e42-proc\") pod \"perf-node-gather-daemonset-62p5v\" (UID: \"d8d6563b-9138-41e4-ab7c-889723489e42\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" Jul 07 15:19:31.619180 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.619139 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8d6563b-9138-41e4-ab7c-889723489e42-sys\") pod \"perf-node-gather-daemonset-62p5v\" (UID: \"d8d6563b-9138-41e4-ab7c-889723489e42\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" Jul 07 15:19:31.619396 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.619231 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8d6563b-9138-41e4-ab7c-889723489e42-podres\") pod \"perf-node-gather-daemonset-62p5v\" (UID: \"d8d6563b-9138-41e4-ab7c-889723489e42\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" Jul 07 15:19:31.619396 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.619287 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8d6563b-9138-41e4-ab7c-889723489e42-lib-modules\") pod \"perf-node-gather-daemonset-62p5v\" (UID: \"d8d6563b-9138-41e4-ab7c-889723489e42\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" Jul 07 15:19:31.619396 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.619314 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfwvw\" (UniqueName: \"kubernetes.io/projected/d8d6563b-9138-41e4-ab7c-889723489e42-kube-api-access-rfwvw\") pod \"perf-node-gather-daemonset-62p5v\" (UID: \"d8d6563b-9138-41e4-ab7c-889723489e42\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" Jul 07 15:19:31.619396 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.619353 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8d6563b-9138-41e4-ab7c-889723489e42-proc\") pod \"perf-node-gather-daemonset-62p5v\" (UID: \"d8d6563b-9138-41e4-ab7c-889723489e42\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" Jul 07 15:19:31.619627 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.619488 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8d6563b-9138-41e4-ab7c-889723489e42-proc\") pod \"perf-node-gather-daemonset-62p5v\" (UID: \"d8d6563b-9138-41e4-ab7c-889723489e42\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" Jul 07 15:19:31.619627 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.619546 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8d6563b-9138-41e4-ab7c-889723489e42-sys\") pod \"perf-node-gather-daemonset-62p5v\" (UID: \"d8d6563b-9138-41e4-ab7c-889723489e42\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" Jul 07 15:19:31.619796 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.619656 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8d6563b-9138-41e4-ab7c-889723489e42-podres\") pod \"perf-node-gather-daemonset-62p5v\" (UID: \"d8d6563b-9138-41e4-ab7c-889723489e42\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" Jul 07 15:19:31.619796 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.619743 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8d6563b-9138-41e4-ab7c-889723489e42-lib-modules\") pod \"perf-node-gather-daemonset-62p5v\" (UID: \"d8d6563b-9138-41e4-ab7c-889723489e42\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" Jul 07 15:19:31.637498 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.628505 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfwvw\" (UniqueName: \"kubernetes.io/projected/d8d6563b-9138-41e4-ab7c-889723489e42-kube-api-access-rfwvw\") pod \"perf-node-gather-daemonset-62p5v\" (UID: \"d8d6563b-9138-41e4-ab7c-889723489e42\") " pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" Jul 07 15:19:31.731801 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.731765 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" Jul 07 15:19:31.779063 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.779033 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/1.log" Jul 07 15:19:31.797643 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.797599 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-x5rws_6c67431a-a8aa-4dcf-89e1-4f6e3c35fa30/console-operator/2.log" Jul 07 15:19:31.909744 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:31.908804 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v"] Jul 07 15:19:32.813436 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:32.812415 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" event={"ID":"d8d6563b-9138-41e4-ab7c-889723489e42","Type":"ContainerStarted","Data":"ec816095e91dea1e8f8aa0d8c9a85fb14ae0c684bcf1909de9e5dead7b0a1116"} Jul 07 15:19:32.813436 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:32.812493 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" event={"ID":"d8d6563b-9138-41e4-ab7c-889723489e42","Type":"ContainerStarted","Data":"ed92f082f2afeb79add618d86e3183a22cebf4768cfe2be5800a4da8571dfee0"} Jul 07 15:19:32.813436 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:32.813311 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" Jul 07 15:19:32.831513 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:32.831460 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" podStartSLOduration=1.831444938 podStartE2EDuration="1.831444938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:19:32.829706752 +0000 UTC m=+2705.740900428" watchObservedRunningTime="2026-07-07 15:19:32.831444938 +0000 UTC m=+2705.742638650" Jul 07 15:19:33.975797 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:33.975733 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tdw9f_e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b/dns/0.log" Jul 07 15:19:33.997440 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:33.997409 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tdw9f_e239d9e6-6a5a-49c2-8b01-a80afbdf4f3b/kube-rbac-proxy/0.log" Jul 07 15:19:34.050247 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:34.050216 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rzkvv_1f877ac1-ec8e-437e-a186-28b499cf63e1/dns-node-resolver/0.log" Jul 07 15:19:34.585850 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:34.585798 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6d68c6fc7-8cmwn_66c9385d-7cec-4bce-b51f-2affa3e0d21a/registry/0.log" Jul 07 15:19:34.616645 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:34.616617 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h78mx_80308b12-3ccc-477c-9578-1ac78d11db32/node-ca/0.log" Jul 07 15:19:35.842947 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:35.842920 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f6b8bcfd-vpbmb_e28b42f3-a1c2-4fa1-9479-b20be56f35c3/kube-auth-proxy/0.log" Jul 07 15:19:35.877642 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:35.877613 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-rsgc7_ffb40f68-9d64-4f10-ad88-68cfb5ec23ba/istio-proxy/0.log" Jul 07 15:19:35.911715 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:35.911680 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-rsgc7_ffb40f68-9d64-4f10-ad88-68cfb5ec23ba/istio-proxy/1.log" Jul 07 15:19:36.692941 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:36.692912 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7dfs5_288a2a6a-6435-4eab-bb18-ff6cf90ec0be/serve-healthcheck-canary/0.log" Jul 07 15:19:37.324303 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:37.324276 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6944g_7de534e8-1ede-4999-9322-2944033cf5be/kube-rbac-proxy/0.log" Jul 07 15:19:37.346258 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:37.346225 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6944g_7de534e8-1ede-4999-9322-2944033cf5be/exporter/0.log" Jul 07 15:19:37.368655 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:37.368631 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6944g_7de534e8-1ede-4999-9322-2944033cf5be/extractor/0.log" Jul 07 15:19:39.697063 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:39.697022 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29723940-rd9nj_835568d6-74c8-48f0-8e21-e4550d1a45d1/cleanup/0.log" Jul 07 15:19:39.832491 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:39.832464 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-psm49/perf-node-gather-daemonset-62p5v" Jul 07 15:19:39.854582 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:39.854554 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-9t5l7_7a82903d-7b22-4a68-bc89-4d2b06b7d5d4/server/0.log" Jul 07 15:19:39.937211 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:39.937177 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-87x9g_5a37d477-1a1c-4084-8b6b-735c1e32a091/manager/0.log" Jul 07 15:19:40.125938 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:40.125904 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-655bb48ff6-rjktz_239c61dc-8cbd-4bca-8407-0f25210678b2/manager/0.log" Jul 07 15:19:40.147854 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:40.147824 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-655bb48ff6-rjktz_239c61dc-8cbd-4bca-8407-0f25210678b2/copy-manifests/0.log" Jul 07 15:19:41.205479 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:41.205447 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f6b8bcfd-vpbmb_e28b42f3-a1c2-4fa1-9479-b20be56f35c3/kube-auth-proxy/0.log" Jul 07 15:19:41.232626 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:41.232593 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-rsgc7_ffb40f68-9d64-4f10-ad88-68cfb5ec23ba/istio-proxy/0.log" Jul 07 15:19:41.270077 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:41.270040 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-rsgc7_ffb40f68-9d64-4f10-ad88-68cfb5ec23ba/istio-proxy/1.log" Jul 07 15:19:41.639478 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:41.639448 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6d85f8474c-9wnls_1744a35d-94a1-40f5-91fc-0e6feb95cc29/manager/0.log" Jul 07 15:19:42.802271 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:42.802242 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f6b8bcfd-vpbmb_e28b42f3-a1c2-4fa1-9479-b20be56f35c3/kube-auth-proxy/0.log" Jul 07 15:19:42.823370 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:42.823340 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-rsgc7_ffb40f68-9d64-4f10-ad88-68cfb5ec23ba/istio-proxy/0.log" Jul 07 15:19:42.857468 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:42.857443 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-rsgc7_ffb40f68-9d64-4f10-ad88-68cfb5ec23ba/istio-proxy/1.log" Jul 07 15:19:48.204350 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:48.204323 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pn6lw_a8945047-a8e7-4958-9133-a5af6e9ac2b1/kube-multus-additional-cni-plugins/0.log" Jul 07 15:19:48.227429 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:48.227396 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pn6lw_a8945047-a8e7-4958-9133-a5af6e9ac2b1/egress-router-binary-copy/0.log" Jul 07 15:19:48.249791 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:48.249769 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pn6lw_a8945047-a8e7-4958-9133-a5af6e9ac2b1/cni-plugins/0.log" Jul 07 15:19:48.272948 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:48.272929 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pn6lw_a8945047-a8e7-4958-9133-a5af6e9ac2b1/bond-cni-plugin/0.log" Jul 07 15:19:48.293001 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:48.292971 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pn6lw_a8945047-a8e7-4958-9133-a5af6e9ac2b1/routeoverride-cni/0.log" Jul 07 15:19:48.315133 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:48.315112 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pn6lw_a8945047-a8e7-4958-9133-a5af6e9ac2b1/whereabouts-cni-bincopy/0.log" Jul 07 15:19:48.336221 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:48.336200 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pn6lw_a8945047-a8e7-4958-9133-a5af6e9ac2b1/whereabouts-cni/0.log" Jul 07 15:19:48.400609 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:48.400582 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jd2vc_4b602982-27d1-4239-b9c7-7e09013cf3d5/kube-multus/0.log" Jul 07 15:19:48.537068 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:48.537038 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xsfqs_d60cabb9-29c9-4f9d-b147-2ca1844f5d96/network-metrics-daemon/0.log" Jul 07 15:19:48.558329 ip-10-0-142-144 kubenswrapper[2583]: I0707 15:19:48.558306 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xsfqs_d60cabb9-29c9-4f9d-b147-2ca1844f5d96/kube-rbac-proxy/0.log"