Jul 06 19:30:32.222935 ip-10-0-128-200 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 06 19:30:32.222945 ip-10-0-128-200 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 06 19:30:32.222955 ip-10-0-128-200 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 06 19:30:32.223386 ip-10-0-128-200 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 06 19:30:42.288527 ip-10-0-128-200 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 06 19:30:42.288549 ip-10-0-128-200 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6c36b727b28544f29951f9e1862109aa --
Jul 06 19:32:31.355738 ip-10-0-128-200 systemd[1]: Starting Kubernetes Kubelet...
Jul 06 19:32:31.705944 ip-10-0-128-200 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 19:32:31.705944 ip-10-0-128-200 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 06 19:32:31.705944 ip-10-0-128-200 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 19:32:31.705944 ip-10-0-128-200 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 06 19:32:31.705944 ip-10-0-128-200 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 19:32:31.707447 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.707381    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 06 19:32:31.711759 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711743    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 19:32:31.711759 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711758    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 19:32:31.711829 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711762    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 19:32:31.711829 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711766    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 19:32:31.711829 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711788    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 19:32:31.711829 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711802    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 19:32:31.711829 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711807    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 19:32:31.711829 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711813    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 19:32:31.711829 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711817    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 19:32:31.711829 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711820    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 19:32:31.711829 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711823    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 19:32:31.711829 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711826    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 19:32:31.711829 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711828    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 19:32:31.711829 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711832    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 19:32:31.711829 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711835    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711838    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711847    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711850    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711853    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711856    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711859    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711864    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711866    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711869    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711872    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711874    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711877    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711879    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711882    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711885    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711889    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711892    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711894    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711897    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 19:32:31.712129 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711899    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711902    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711904    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711907    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711910    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711912    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711915    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711917    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711920    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711922    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711925    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711927    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711930    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711932    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711936    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711939    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711941    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711944    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711947    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711949    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 19:32:31.712605 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711952    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711954    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711957    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711959    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711962    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711964    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711967    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711969    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711972    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711974    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711977    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711979    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711982    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711984    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711989    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711992    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711994    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711997    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.711999    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712002    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 19:32:31.713101 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712004    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 19:32:31.713575 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712007    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 19:32:31.713575 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712009    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 19:32:31.713575 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712012    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 19:32:31.713575 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712014    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 19:32:31.713575 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712017    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 19:32:31.713575 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712022    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 19:32:31.713575 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712026    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 19:32:31.713575 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712030    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 19:32:31.713575 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712033    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 19:32:31.713575 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712036    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 19:32:31.713575 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712038    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 19:32:31.713575 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712041    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 19:32:31.713575 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712043    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 19:32:31.713575 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712046    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 19:32:31.713575 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712049    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 19:32:31.713575 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712051    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 19:32:31.713575 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712054    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 19:32:31.713575 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712056    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 19:32:31.713575 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712059    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712061    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712064    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712066    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712069    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712072    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712811    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712817    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712820    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712823    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712826    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712828    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712831    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712834    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712836    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712839    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712841    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712843    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712847    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712850    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 19:32:31.714048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712852    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 19:32:31.714540 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712855    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 19:32:31.714540 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712859    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 19:32:31.714540 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712862    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 19:32:31.714540 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712864    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 19:32:31.714540 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712866    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 19:32:31.714540 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712871    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 19:32:31.714540 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712875    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 19:32:31.714540 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712878    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 19:32:31.714540 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712881    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 19:32:31.714540 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712885    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 19:32:31.714540 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712888    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 19:32:31.714540 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712892    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 19:32:31.714540 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712894    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 19:32:31.714540 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712897    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 19:32:31.714540 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712899    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 19:32:31.714540 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712902    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 19:32:31.714540 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712905    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 19:32:31.714540 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712907    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 19:32:31.714540 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712910    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712912    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712915    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712917    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712919    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712922    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712925    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712927    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712930    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712933    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712936    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712939    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712942    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712944    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712947    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712949    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712952    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712955    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712957    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712960    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 19:32:31.715017 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712962    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712964    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712967    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712969    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712971    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712974    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712976    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712979    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712981    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712984    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712986    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712989    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712991    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712993    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712997    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.712999    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713002    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713004    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713007    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713009    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 19:32:31.715520 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713011    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713014    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713016    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713020    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713023    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713025    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713028    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713031    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713033    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713036    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713038    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713042    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713046    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713048    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713051    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713054    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713057    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713060    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713062    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713064    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 19:32:31.716007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713067    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713070    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713072    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713075    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713077    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713080    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713411    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713415    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713418    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713421    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713423    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713426    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713428    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713431    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713434    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713436    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713441    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713444    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713447    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713449    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 19:32:31.716492 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713452    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713455    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713457    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713460    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713462    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713464    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713467    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713469    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713472    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713474    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713477    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713479    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713498    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713501    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713503    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713506    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713508    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713511    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713514    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713516    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 19:32:31.716972 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713521    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 19:32:31.717451 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713524    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 19:32:31.717451 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713527    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 19:32:31.717451 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713530    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 19:32:31.717451 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713534    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 19:32:31.717451 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713536    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 19:32:31.717451 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713539    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 19:32:31.717451 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713541    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 19:32:31.717451 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713546    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 19:32:31.717451 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713548    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 19:32:31.717451 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713551    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 19:32:31.717451 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713553    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 19:32:31.717451 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713556    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 19:32:31.717451 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713559    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 19:32:31.717451 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713562    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 19:32:31.717451 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713564    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 19:32:31.717451 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713567    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 19:32:31.717451 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713569    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 19:32:31.717451 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713571    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 19:32:31.717451 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713575    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713579    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713582    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713585    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713589    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713593    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713595    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713598    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713601    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713603    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713606    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713608    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713611    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713613    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713615    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713618    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713620    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713623    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713625    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713628    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 19:32:31.717903 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713630    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713633    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713636    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713638    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713641    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713644    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713646    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713649    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713652    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713655    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713657    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713659    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713662    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713664    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713666    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713669    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713671    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713674    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713676    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713678    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 19:32:31.718390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713681    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713683    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713686    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713690    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713692    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.713695    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713754    2562 flags.go:64] FLAG: --address="0.0.0.0"
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713760    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713766    2562 flags.go:64] FLAG: --anonymous-auth="true"
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713770    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713774    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713777    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713781    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713791    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713794    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713798    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713801    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713804    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713807    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713810    2562 flags.go:64] FLAG: --cgroup-root=""
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713813    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713816    2562 flags.go:64] FLAG: --client-ca-file=""
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713819    2562 flags.go:64] FLAG: --cloud-provider="external"
Jul 06 19:32:31.718887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713822    2562 flags.go:64] FLAG: --cluster-dns="[]"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713826    2562 flags.go:64] FLAG: --cluster-domain=""
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713828    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713831    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713834    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713837    2562 flags.go:64] FLAG: --container-log-max-files="5"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713841    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713843    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713846    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713850    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713853    2562 flags.go:64] FLAG: --contention-profiling="false"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713855    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713858    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713861    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713864    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713868    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713871    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713873    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713876    2562 flags.go:64] FLAG: --enable-load-reader="false"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713879    2562 flags.go:64] FLAG: --enable-server="true"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713881    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713885    2562 flags.go:64] FLAG: --event-burst="100"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713888    2562 flags.go:64] FLAG: --event-qps="50"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713891    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 06 19:32:31.719431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713894    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713897    2562 flags.go:64] FLAG: --eviction-hard=""
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713901    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713904    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713907    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713910    2562 flags.go:64] FLAG: --eviction-soft=""
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713913    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713916    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713919    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713922    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713924    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713927    2562 flags.go:64] FLAG: --fail-swap-on="true"
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713930    2562 flags.go:64] FLAG: --feature-gates=""
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713933    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713936    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713939    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713943    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713945    2562 flags.go:64] FLAG: --healthz-port="10248"
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713948    2562 flags.go:64] FLAG: --help="false"
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713951    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-128-200.ec2.internal"
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713954    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713957    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713959    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713963    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 06 19:32:31.720046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713966    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713969    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713971    2562 flags.go:64] FLAG: --image-service-endpoint=""
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713974    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713977    2562 flags.go:64] FLAG: --kube-api-burst="100"
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713979    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713983    2562 flags.go:64] FLAG: --kube-api-qps="50"
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713985    2562 flags.go:64] FLAG: --kube-reserved=""
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713988    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713991    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713994    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713996    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.713999    2562 flags.go:64] FLAG: --lock-file=""
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714001    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714004    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714007    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714013    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714015    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714018    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714021    2562 flags.go:64] FLAG: --logging-format="text"
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714023    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714026    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714029    2562 flags.go:64] FLAG: --manifest-url=""
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714032    2562 flags.go:64] FLAG: --manifest-url-header=""
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714035    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 06 19:32:31.721016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714038    2562 flags.go:64] FLAG: --max-open-files="1000000"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714042    2562 flags.go:64] FLAG: --max-pods="110"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714045    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714048    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714051    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714053    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714057    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714060    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714062    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714069    2562 flags.go:64] FLAG: --node-status-max-images="50"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714072    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714075    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714078    2562 flags.go:64] FLAG: --pod-cidr=""
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714082    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714087    2562 flags.go:64] FLAG: --pod-manifest-path=""
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714089    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714094    2562 flags.go:64] FLAG: --pods-per-core="0"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714097    2562 flags.go:64] FLAG: --port="10250"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714100    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714103    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d4d5907e99d0834b"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714106    2562 flags.go:64] FLAG: --qos-reserved=""
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714109    2562 flags.go:64] FLAG: --read-only-port="10255"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714112    2562 flags.go:64] FLAG: --register-node="true"
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714114    2562 flags.go:64] FLAG: --register-with-taints=""
Jul 06 19:32:31.722098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714118    2562 flags.go:64] FLAG: --registry-burst="10"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714121    2562 flags.go:64] FLAG: --registry-qps="5"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714124    2562 flags.go:64] FLAG: --reserved-cpus=""
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714127    2562 flags.go:64] FLAG: --reserved-memory=""
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714130    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714133    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714136    2562 flags.go:64] FLAG: --rotate-certificates="false"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714138    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714141    2562 flags.go:64] FLAG: --runonce="false"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714144    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714147    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714150    2562 flags.go:64] FLAG: --seccomp-default="false"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714152    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714155    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714158    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714161    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714164    2562 flags.go:64] FLAG: --storage-driver-password="root"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714167    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714169    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714172    2562 flags.go:64] FLAG: --storage-driver-user="root"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714175    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714178    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714182    2562 flags.go:64] FLAG: --system-cgroups=""
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714185    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714192    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 06 19:32:31.723137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714194    2562 flags.go:64] FLAG: --tls-cert-file=""
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714197    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714200    2562 flags.go:64] FLAG: --tls-min-version=""
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714203    2562 flags.go:64] FLAG: --tls-private-key-file=""
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714206    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714209    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714212    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714214    2562 flags.go:64] FLAG: --v="2"
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714219    2562 flags.go:64] FLAG: --version="false"
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714222    2562 flags.go:64] FLAG: --vmodule=""
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714226    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.714229    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714343    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714347    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714350    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714353    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714356    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714358    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714360    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714363    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714366    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714368    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 19:32:31.724316 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714371    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714373    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714376    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714378    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714381    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714383    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714385    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714391    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714394    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714397    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714400    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714402    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714405    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714407    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714410    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714412    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714415    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714417    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714420    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714424    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 19:32:31.725236 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714426    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 19:32:31.725803 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714428    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 19:32:31.725803 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714431    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 19:32:31.725803 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714433    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 19:32:31.725803 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714436    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 19:32:31.725803 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714439    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 19:32:31.725803 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714441    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 19:32:31.725803 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714444    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 19:32:31.725803 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714446    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 19:32:31.725803 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714449    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 19:32:31.725803 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714451    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 19:32:31.725803 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714454    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 19:32:31.725803 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714456    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 19:32:31.725803 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714459    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 19:32:31.725803 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714480    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 19:32:31.725803 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714497    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 19:32:31.725803 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714500    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 19:32:31.725803 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714503    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 19:32:31.725803 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714507    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 19:32:31.725803 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714512    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714515    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714519    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714521    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714524    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714526    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714529    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714531    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714534    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714536    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714539    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714541    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714544    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714547    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714550    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714553    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714556    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714559    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714563    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714565    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 19:32:31.726245 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714568    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714571    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714574    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714578    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714581    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714584    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714587    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714590    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714592    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714595    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714597    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714600    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714604    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714607    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714611    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714613    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714616    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714618    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714621    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714623    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 19:32:31.726757 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714626    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 19:32:31.727234 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714628    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 19:32:31.727234 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714631    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 19:32:31.727234 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714633    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 19:32:31.727234 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714636    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 19:32:31.727234 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714638    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 19:32:31.727234 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714642    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 19:32:31.727234 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714644    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 19:32:31.727234 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714647    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 19:32:31.727234 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.714649    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 19:32:31.727234 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.715279    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 19:32:31.727234 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.723125    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 06 19:32:31.727234 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.723143    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 06 19:32:31.727234 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.723169    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 06 19:32:31.727234 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.723182    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 06 19:32:31.727234 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723273    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723280    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723284    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723309    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723315    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723320    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723325    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723330    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723335    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723339    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723343    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723348    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723352    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723356    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723361    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723365    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723369    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723373    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723377    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723381    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 19:32:31.727609 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723385    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 19:32:31.728076 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723389    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 19:32:31.728076 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723395    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 19:32:31.728076 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723409    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 19:32:31.728076 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723415    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 19:32:31.728076 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723419    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 19:32:31.728076 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723425    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 19:32:31.728076 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723431    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 19:32:31.728076 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723436    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 19:32:31.728076 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723442    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 19:32:31.728076 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723447    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 19:32:31.728076 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723451    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 19:32:31.728076 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723456    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 19:32:31.728076 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723460    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 19:32:31.728076 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723464    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 19:32:31.728076 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723468    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 19:32:31.728076 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723474    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 19:32:31.728076 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723480    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 19:32:31.728076 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723504    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 19:32:31.728076 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723508    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723513    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723518    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723522    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723526    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723530    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723534    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723538    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723542    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723546    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723551    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723555    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723559    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723563    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723567    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723571    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723575    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723587    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723592    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723596    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 19:32:31.728560 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723600    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723605    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723610    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723614    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723618    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723623    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723627    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723631    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723635    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723640    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723645    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723649    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723653    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723658    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723662    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723666    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723670    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723674    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723678    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723682    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 19:32:31.729050 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723687    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 19:32:31.729556 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723690    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 19:32:31.729556 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723695    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 19:32:31.729556 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723699    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 19:32:31.729556 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723703    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 19:32:31.729556 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723708    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 19:32:31.729556 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723712    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 19:32:31.729556 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723716    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 19:32:31.729556 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723720    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 19:32:31.729556 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723724    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 19:32:31.729556 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723735    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 19:32:31.729556 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723740    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 19:32:31.729556 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723744    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 19:32:31.729556 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723748    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 19:32:31.729556 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723753    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 19:32:31.729556 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723757    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 19:32:31.729556 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723761    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 19:32:31.729556 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723765    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 19:32:31.729556 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723770    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 19:32:31.729556 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.723774    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 19:32:31.730013 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.723782    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 19:32:31.730013 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725396    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 19:32:31.730013 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725409    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 19:32:31.730013 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725413    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 19:32:31.730013 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725416    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 19:32:31.730013 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725419    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 19:32:31.730013 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725422    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 19:32:31.730013 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725424    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 19:32:31.730013 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725427    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 19:32:31.730013 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725430    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 19:32:31.730013 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725433    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 19:32:31.730013 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725437    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 19:32:31.730013 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725460    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 19:32:31.730013 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725464    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 19:32:31.730013 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725467    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725470    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725474    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725477    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725479    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725498    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725503    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725506    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725509    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725512    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725515    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725517    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725520    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725523    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725525    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725528    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725531    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725533    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725537    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725540    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 19:32:31.730390 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725543    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725546    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725549    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725551    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725554    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725556    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725559    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725561    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725564    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725566    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725569    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725571    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725574    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725576    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725578    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725581    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725583    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725588    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725591    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725593    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 19:32:31.730900 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725596    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 19:32:31.731382 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725598    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 19:32:31.731382 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725601    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 19:32:31.731382 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725603    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 19:32:31.731382 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725606    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 19:32:31.731382 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725610    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 19:32:31.731382 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725614    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 19:32:31.731382 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725617    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 19:32:31.731382 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725620    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 19:32:31.731382 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725623    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 19:32:31.731382 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725626    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 19:32:31.731382 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725629    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 19:32:31.731382 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725633    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 19:32:31.731382 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725636    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 19:32:31.731382 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725639    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 19:32:31.731382 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725642    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 19:32:31.731382 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725645    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 19:32:31.731382 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725648    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 19:32:31.731382 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725650    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 19:32:31.731382 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725653    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 19:32:31.731886 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725655    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 19:32:31.731886 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725658    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 19:32:31.731886 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725660    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 19:32:31.731886 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725663    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 19:32:31.731886 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725665    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 19:32:31.731886 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725668    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 19:32:31.731886 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725670    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 19:32:31.731886 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725672    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 19:32:31.731886 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725675    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 19:32:31.731886 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725677    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 19:32:31.731886 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725681    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 19:32:31.731886 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725685    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 19:32:31.731886 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725687    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 19:32:31.731886 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725690    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 19:32:31.731886 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725693    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 19:32:31.731886 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725695    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 19:32:31.731886 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725698    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 19:32:31.731886 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725700    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 19:32:31.731886 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725703    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 19:32:31.732344 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725705    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 19:32:31.732344 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725708    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 19:32:31.732344 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725710    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 19:32:31.732344 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725712    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 19:32:31.732344 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725715    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 19:32:31.732344 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725719    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 19:32:31.732344 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:31.725723    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 19:32:31.732344 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.725728    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 19:32:31.732344 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.725829    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 06 19:32:31.732344 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.727941    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 06 19:32:31.732344 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.729375    2562 server.go:1019] "Starting client certificate rotation"
Jul 06 19:32:31.732344 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.729469    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 19:32:31.732344 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.729512    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 19:32:31.732736 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.731711    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 19:32:31.735883 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.735869    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 19:32:31.745962 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.745945    2562 log.go:25] "Validated CRI v1 runtime API"
Jul 06 19:32:31.748400 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.748386    2562 log.go:25] "Validated CRI v1 image API"
Jul 06 19:32:31.749907 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.749885    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 06 19:32:31.752473 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.752453    2562 fs.go:135] Filesystem UUIDs: map[4fde1e68-8ef7-4f59-b8c9-506e2fc79b74:/dev/nvme0n1p3 6e27127a-5411-405c-aeed-6cfb8e720577:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 06 19:32:31.752551 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.752474    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 06 19:32:31.759195 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.759095    2562 manager.go:216] Machine: {Timestamp:2026-07-06 19:32:31.757887503 +0000 UTC m=+0.299064169 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100951 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2575b65221c8f5bac2e29a9461fd4b SystemUUID:ec2575b6-5221-c8f5-bac2-e29a9461fd4b BootID:6c36b727-b285-44f2-9951-f9e1862109aa Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ee:89:6e:bc:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ee:89:6e:bc:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:8d:d2:ac:ec:39 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 06 19:32:31.759195 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.759193    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 06 19:32:31.759292 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.759259    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 06 19:32:31.761781 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.761764    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 19:32:31.762243 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.762218    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 06 19:32:31.762384 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.762248    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-200.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 06 19:32:31.763567 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.763558    2562 topology_manager.go:138] "Creating topology manager with none policy"
Jul 06 19:32:31.763605 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.763570    2562 container_manager_linux.go:309] "Creating device plugin manager"
Jul 06 19:32:31.763605 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.763577    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 19:32:31.764220 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.764211    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 19:32:31.765634 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.765624    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 06 19:32:31.769879 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.769868    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 19:32:31.770110 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.770101    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 06 19:32:31.770139 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.770129    2562 kubelet.go:486] "Attempting to sync node with API server"
Jul 06 19:32:31.770168 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.770144    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 06 19:32:31.770168 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.770157    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 06 19:32:31.770221 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.770169    2562 kubelet.go:398] "Adding apiserver pod source"
Jul 06 19:32:31.770221 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.770200    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 06 19:32:31.770766 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.770733    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 19:32:31.770766 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.770763    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 19:32:31.773610 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.773587    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 06 19:32:31.775677 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.775657    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 06 19:32:31.775757 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.775700    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 06 19:32:31.777012 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.776999    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 06 19:32:31.777079 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.777019    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 06 19:32:31.777079 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.777027    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 06 19:32:31.777079 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.777036    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 06 19:32:31.777079 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.777049    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 06 19:32:31.777079 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.777058    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 06 19:32:31.777079 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.777070    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 06 19:32:31.777079 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.777077    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 06 19:32:31.777300 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.777087    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 06 19:32:31.777300 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.777095    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 06 19:32:31.777300 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.777107    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 06 19:32:31.777300 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.777115    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 06 19:32:31.778538 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.778527    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 06 19:32:31.778651 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.778541    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 06 19:32:31.781425 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.781410    2562 server.go:1268] "Started kubelet"
Jul 06 19:32:31.781559 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.781510    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 06 19:32:31.781648 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.781512    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 06 19:32:31.781648 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.781579    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 06 19:32:31.782220 ip-10-0-128-200 systemd[1]: Started Kubernetes Kubelet.
Jul 06 19:32:31.783400 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.783380    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 06 19:32:31.784065 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.784048    2562 server.go:310] "Adding debug handlers to kubelet server"
Jul 06 19:32:31.784220 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:31.784194    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-200.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 06 19:32:31.784317 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:31.784272    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 06 19:32:31.784429 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.784409    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-200.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 06 19:32:31.787269 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:31.787242    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 06 19:32:31.789610 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.789597    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 06 19:32:31.789667 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.789615    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 06 19:32:31.789735 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:31.789676    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-200.ec2.internal\" not found"
Jul 06 19:32:31.790223 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.790203    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 06 19:32:31.790322 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.790229    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 06 19:32:31.790322 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.790203    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 06 19:32:31.790415 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.790382    2562 reconstruct.go:97] "Volume reconstruction finished"
Jul 06 19:32:31.790415 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.790394    2562 reconciler.go:29] "Reconciler: start to sync state"
Jul 06 19:32:31.790517 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.790415    2562 factory.go:55] Registering systemd factory
Jul 06 19:32:31.790517 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.790427    2562 factory.go:223] Registration of the systemd container factory successfully
Jul 06 19:32:31.790730 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.790703    2562 factory.go:153] Registering CRI-O factory
Jul 06 19:32:31.790730 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.790727    2562 factory.go:223] Registration of the crio container factory successfully
Jul 06 19:32:31.790879 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.790790    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 06 19:32:31.790879 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.790815    2562 factory.go:103] Registering Raw factory
Jul 06 19:32:31.790879 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.790835    2562 manager.go:1196] Started watching for new ooms in manager
Jul 06 19:32:31.792545 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.791233    2562 manager.go:349] Starting recovery of all containers
Jul 06 19:32:31.792545 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:31.792169    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-200.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 06 19:32:31.792545 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:31.792301    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 06 19:32:31.792881 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:31.791983    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-200.ec2.internal.18bfca3f8e5050dd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-200.ec2.internal,UID:ip-10-0-128-200.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-200.ec2.internal,},FirstTimestamp:2026-07-06 19:32:31.781384413 +0000 UTC m=+0.322561080,LastTimestamp:2026-07-06 19:32:31.781384413 +0000 UTC m=+0.322561080,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-200.ec2.internal,}"
Jul 06 19:32:31.801732 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.801635    2562 manager.go:354] Recovery completed
Jul 06 19:32:31.805766 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.805754    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 19:32:31.809388 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.809361    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 19:32:31.809453 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.809404    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 19:32:31.809453 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.809419    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasSufficientPID"
Jul 06 19:32:31.809979 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.809964    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 06 19:32:31.809979 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.809979    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 06 19:32:31.810113 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.809969    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 19:32:31.810113 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.809997    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 19:32:31.810634 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.810620    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 19:32:31.810686 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.810645    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 19:32:31.810686 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.810659    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasSufficientPID"
Jul 06 19:32:31.812371 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.812359    2562 policy_none.go:49] "None policy: Start"
Jul 06 19:32:31.812431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.812377    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 06 19:32:31.812431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.812389    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 06 19:32:31.814561 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.814550    2562 policy_none.go:47] "Start"
Jul 06 19:32:31.815880 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.815865    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z6lrs"
Jul 06 19:32:31.820857 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:31.820795    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-200.ec2.internal.18bfca3f8ffb9b22  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-200.ec2.internal,UID:ip-10-0-128-200.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-200.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-200.ec2.internal,},FirstTimestamp:2026-07-06 19:32:31.809387298 +0000 UTC m=+0.350563973,LastTimestamp:2026-07-06 19:32:31.809387298 +0000 UTC m=+0.350563973,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-200.ec2.internal,}"
Jul 06 19:32:31.831654 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:31.831588    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-200.ec2.internal.18bfca3f8ffbfa7f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-200.ec2.internal,UID:ip-10-0-128-200.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-200.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-200.ec2.internal,},FirstTimestamp:2026-07-06 19:32:31.809411711 +0000 UTC m=+0.350588587,LastTimestamp:2026-07-06 19:32:31.809411711 +0000 UTC m=+0.350588587,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-200.ec2.internal,}"
Jul 06 19:32:31.840751 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:31.840690    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-200.ec2.internal.18bfca3f8ffc312b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-200.ec2.internal,UID:ip-10-0-128-200.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-200.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-200.ec2.internal,},FirstTimestamp:2026-07-06 19:32:31.809425707 +0000 UTC m=+0.350602373,LastTimestamp:2026-07-06 19:32:31.809425707 +0000 UTC m=+0.350602373,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-200.ec2.internal,}"
Jul 06 19:32:31.846518 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.846500    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z6lrs"
Jul 06 19:32:31.853012 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.853000    2562 manager.go:337] "Starting Device Plugin manager"
Jul 06 19:32:31.853099 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:31.853025    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 06 19:32:31.853099 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.853033    2562 server.go:87] "Starting device plugin registration server"
Jul 06 19:32:31.853235 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.853224    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 06 19:32:31.853270 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.853237    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 06 19:32:31.853334 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.853315    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 06 19:32:31.853416 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.853400    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 06 19:32:31.853416 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.853413    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 06 19:32:31.853919 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:31.853901    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 06 19:32:31.853991 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:31.853947    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-200.ec2.internal\" not found"
Jul 06 19:32:31.887193 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.887173    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 06 19:32:31.895402 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.888312    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 06 19:32:31.895402 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.888343    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 06 19:32:31.895402 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.888363    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 06 19:32:31.895402 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:31.888393    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 06 19:32:31.895402 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.890761    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 19:32:31.953908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.953882    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 19:32:31.954754 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.954739    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 19:32:31.954839 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.954770    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 19:32:31.954839 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.954783    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasSufficientPID"
Jul 06 19:32:31.954839 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.954810    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-200.ec2.internal"
Jul 06 19:32:31.965288 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.965246    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-200.ec2.internal"
Jul 06 19:32:31.965288 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:31.965266    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-200.ec2.internal\": node \"ip-10-0-128-200.ec2.internal\" not found"
Jul 06 19:32:31.980161 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:31.980141    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-200.ec2.internal\" not found"
Jul 06 19:32:31.989809 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.989791    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-200.ec2.internal"]
Jul 06 19:32:31.989858 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.989840    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 19:32:31.990539 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.990523    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 19:32:31.990604 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.990553    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 19:32:31.990604 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.990563    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasSufficientPID"
Jul 06 19:32:31.990692 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.990612    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cd31cf43040b67356073a3b42229e6bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal\" (UID: \"cd31cf43040b67356073a3b42229e6bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal"
Jul 06 19:32:31.990692 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.990633    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd31cf43040b67356073a3b42229e6bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal\" (UID: \"cd31cf43040b67356073a3b42229e6bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal"
Jul 06 19:32:31.992818 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.992802    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 19:32:31.992944 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.992929    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal"
Jul 06 19:32:31.992987 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.992962    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 19:32:31.993498 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.993467    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 19:32:31.993585 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.993503    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 19:32:31.993585 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.993533    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasSufficientPID"
Jul 06 19:32:31.993585 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.993542    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 19:32:31.993585 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.993562    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 19:32:31.993585 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.993573    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasSufficientPID"
Jul 06 19:32:31.995848 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.995831    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-200.ec2.internal"
Jul 06 19:32:31.995923 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.995874    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 19:32:31.996465 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.996447    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 19:32:31.996561 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.996475    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 19:32:31.996561 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:31.996503    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeHasSufficientPID"
Jul 06 19:32:32.018985 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:32.018963    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-200.ec2.internal\" not found" node="ip-10-0-128-200.ec2.internal"
Jul 06 19:32:32.023066 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:32.023048    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-200.ec2.internal\" not found" node="ip-10-0-128-200.ec2.internal"
Jul 06 19:32:32.081055 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:32.081035    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-200.ec2.internal\" not found"
Jul 06 19:32:32.091410 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.091384    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cd31cf43040b67356073a3b42229e6bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal\" (UID: \"cd31cf43040b67356073a3b42229e6bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal"
Jul 06 19:32:32.091517 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.091414    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd31cf43040b67356073a3b42229e6bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal\" (UID: \"cd31cf43040b67356073a3b42229e6bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal"
Jul 06 19:32:32.091517 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.091444    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd31cf43040b67356073a3b42229e6bd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal\" (UID: \"cd31cf43040b67356073a3b42229e6bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal"
Jul 06 19:32:32.091517 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.091476    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cd31cf43040b67356073a3b42229e6bd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal\" (UID: \"cd31cf43040b67356073a3b42229e6bd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal"
Jul 06 19:32:32.183059 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:32.183024    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-200.ec2.internal\" not found"
Jul 06 19:32:32.192016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.191994    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc01bcdac46728d38f70a288423f3cc9-config\") pod \"kube-apiserver-proxy-ip-10-0-128-200.ec2.internal\" (UID: \"bc01bcdac46728d38f70a288423f3cc9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-200.ec2.internal"
Jul 06 19:32:32.283283 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:32.283227    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-200.ec2.internal\" not found"
Jul 06 19:32:32.292226 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.292200    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc01bcdac46728d38f70a288423f3cc9-config\") pod \"kube-apiserver-proxy-ip-10-0-128-200.ec2.internal\" (UID: \"bc01bcdac46728d38f70a288423f3cc9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-200.ec2.internal"
Jul 06 19:32:32.292291 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.292241    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc01bcdac46728d38f70a288423f3cc9-config\") pod \"kube-apiserver-proxy-ip-10-0-128-200.ec2.internal\" (UID: \"bc01bcdac46728d38f70a288423f3cc9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-200.ec2.internal"
Jul 06 19:32:32.323702 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.323676    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal"
Jul 06 19:32:32.326849 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.326833    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-200.ec2.internal"
Jul 06 19:32:32.383637 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:32.383606    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-200.ec2.internal\" not found"
Jul 06 19:32:32.485672 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:32.485644    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-200.ec2.internal\" not found"
Jul 06 19:32:32.586403 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:32.586335    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-200.ec2.internal\" not found"
Jul 06 19:32:32.686554 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:32.686523    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-200.ec2.internal\" not found"
Jul 06 19:32:32.713569 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.713542    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 19:32:32.728884 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.728859    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 06 19:32:32.729017 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.728992    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 19:32:32.729062 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.729019    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 19:32:32.771152 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.771126    2562 apiserver.go:52] "Watching apiserver"
Jul 06 19:32:32.790596 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.790554    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 06 19:32:32.790718 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.790555    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-200.ec2.internal"
Jul 06 19:32:32.797223 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.797195    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 19:32:32.798835 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.798813    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 06 19:32:32.799192 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.799167    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-tkgjn","openshift-multus/network-metrics-daemon-kbl7t","kube-system/global-pull-secret-syncer-zgsll","openshift-network-diagnostics/network-check-target-lvxg5","openshift-network-operator/iptables-alerter-z5bg8","openshift-ovn-kubernetes/ovnkube-node-wt558","kube-system/konnectivity-agent-cklnx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt","openshift-cluster-node-tuning-operator/tuned-dfspr","openshift-dns/node-resolver-n9f6w","openshift-image-registry/node-ca-2cwdm","openshift-multus/multus-additional-cni-plugins-5rwdf"]
Jul 06 19:32:32.801693 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.801672    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 19:32:32.801769 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.801754    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal"
Jul 06 19:32:32.801943 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.801926    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.804378 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.804353    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:32.804477 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:32.804423    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvxg5" podUID="2fc30bdf-7515-45f6-9c88-2f976891430f"
Jul 06 19:32:32.804762 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.804743    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 06 19:32:32.804812 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.804743    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 19:32:32.805296 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.805276    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 06 19:32:32.805397 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.805280    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rk54j\""
Jul 06 19:32:32.805639 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.805622    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 06 19:32:32.805697 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.805628    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 06 19:32:32.806896 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.806871    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.809342 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.809323    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cklnx"
Jul 06 19:32:32.810491 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.810473    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 06 19:32:32.810754 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.810726    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tgjz7\""
Jul 06 19:32:32.810849 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.810833    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 06 19:32:32.811766 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.811749    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:32.812164 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.812141    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 06 19:32:32.812251 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.812153    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 06 19:32:32.812309 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.812183    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 06 19:32:32.812800 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.812782    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 06 19:32:32.812961 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.812938    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 06 19:32:32.813156 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.813142    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2mhlq\""
Jul 06 19:32:32.813766 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.813747    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 06 19:32:32.814248 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.814234    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.815453 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.815437    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 06 19:32:32.815857 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.815838    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 06 19:32:32.815914 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.815859    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 06 19:32:32.816360 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.816345    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ww8w7\""
Jul 06 19:32:32.816738 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.816722    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:32.816844 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:32.816820    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbl7t" podUID="2bd4ee01-7358-4e6f-9b67-43d41aef4fc3"
Jul 06 19:32:32.817111 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.817070    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 06 19:32:32.817687 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.817672    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 06 19:32:32.817687 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.817682    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s8vwb\""
Jul 06 19:32:32.819189 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.819170    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:32.819277 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:32.819255    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgsll" podUID="05300884-0a61-4b61-9637-add8ddfff1c5"
Jul 06 19:32:32.819903 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.819882    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 19:32:32.821724 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.821701    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z5bg8"
Jul 06 19:32:32.824150 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.824128    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n9f6w"
Jul 06 19:32:32.824749 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.824731    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 06 19:32:32.825048 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.825032    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 06 19:32:32.825293 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.825275    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 06 19:32:32.825388 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.825304    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pc8r2\""
Jul 06 19:32:32.826653 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.826636    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2cwdm"
Jul 06 19:32:32.826904 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.826881    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 06 19:32:32.827791 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.827756    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 06 19:32:32.827852 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.827802    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bddq6\""
Jul 06 19:32:32.829138 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.829120    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-200.ec2.internal"]
Jul 06 19:32:32.829205 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.829147    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal"]
Jul 06 19:32:32.829247 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.829219    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:32.829722 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.829704    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 06 19:32:32.829777 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.829734    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 06 19:32:32.829841 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.829734    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 06 19:32:32.829841 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.829823    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fc4bl\""
Jul 06 19:32:32.832058 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.832037    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 06 19:32:32.832161 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.832037    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 06 19:32:32.832161 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.832072    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4pfkh\""
Jul 06 19:32:32.835713 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.835693    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m57hh"
Jul 06 19:32:32.849145 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.849103    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 19:27:32 +0000 UTC" deadline="2027-12-24 10:04:06.680359871 +0000 UTC"
Jul 06 19:32:32.849145 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.849138    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12854h31m33.831224599s"
Jul 06 19:32:32.849145 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.849144    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m57hh"
Jul 06 19:32:32.850858 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.850843    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 19:32:32.891657 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.891631    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 06 19:32:32.895803 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.895779    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-run\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.895887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.895819    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-var-lib-kubelet\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.895887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.895870    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-host\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.896012 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.895907    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b29855f2-a8a2-49f4-8d0b-7f52c478e852-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:32.896012 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.895935    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm9gf\" (UniqueName: \"kubernetes.io/projected/75a74486-1003-45bb-9c55-1cbf52ab4090-kube-api-access-gm9gf\") pod \"node-ca-2cwdm\" (UID: \"75a74486-1003-45bb-9c55-1cbf52ab4090\") " pod="openshift-image-registry/node-ca-2cwdm"
Jul 06 19:32:32.896012 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.895960    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-kubelet\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.896012 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.895986    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-ovn-node-metrics-cert\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.896168 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896009    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-host-var-lib-cni-bin\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.896168 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896048    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-host-var-lib-cni-multus\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.896168 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896076    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-multus-conf-dir\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.896168 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896094    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-etc-kubernetes\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.896168 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896109    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-multus-socket-dir-parent\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.896168 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896126    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97xjd\" (UniqueName: \"kubernetes.io/projected/b29855f2-a8a2-49f4-8d0b-7f52c478e852-kube-api-access-97xjd\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:32.896454 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896153    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-slash\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.896454 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896203    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-ovnkube-config\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.896454 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896246    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-lib-modules\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.896454 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896269    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-cnibin\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.896454 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896291    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-cni-binary-copy\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.896454 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896318    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b29855f2-a8a2-49f4-8d0b-7f52c478e852-socket-dir\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:32.896454 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896340    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-log-socket\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.896454 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896355    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-cni-netd\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.896454 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896382    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-env-overrides\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.896454 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896433    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba-agent-certs\") pod \"konnectivity-agent-cklnx\" (UID: \"b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba\") " pod="kube-system/konnectivity-agent-cklnx"
Jul 06 19:32:32.896908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896461    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05300884-0a61-4b61-9637-add8ddfff1c5-kubelet-config\") pod \"global-pull-secret-syncer-zgsll\" (UID: \"05300884-0a61-4b61-9637-add8ddfff1c5\") " pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:32.896908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896489    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztm2q\" (UniqueName: \"kubernetes.io/projected/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-kube-api-access-ztm2q\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.896908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896538    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn4fv\" (UniqueName: \"kubernetes.io/projected/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-kube-api-access-nn4fv\") pod \"network-metrics-daemon-kbl7t\" (UID: \"2bd4ee01-7358-4e6f-9b67-43d41aef4fc3\") " pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:32.896908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896559    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0300b33e-bd11-401e-ac65-312bcd5d72a1-host-slash\") pod \"iptables-alerter-z5bg8\" (UID: \"0300b33e-bd11-401e-ac65-312bcd5d72a1\") " pod="openshift-network-operator/iptables-alerter-z5bg8"
Jul 06 19:32:32.896908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896575    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-host-run-k8s-cni-cncf-io\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.896908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896590    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05300884-0a61-4b61-9637-add8ddfff1c5-dbus\") pod \"global-pull-secret-syncer-zgsll\" (UID: \"05300884-0a61-4b61-9637-add8ddfff1c5\") " pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:32.896908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896611    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea2a94f-253e-4bc7-960f-69e371dbbd6a-tmp-dir\") pod \"node-resolver-n9f6w\" (UID: \"5ea2a94f-253e-4bc7-960f-69e371dbbd6a\") " pod="openshift-dns/node-resolver-n9f6w"
Jul 06 19:32:32.896908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896627    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-system-cni-dir\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.896908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896647    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b29855f2-a8a2-49f4-8d0b-7f52c478e852-sys-fs\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:32.896908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896678    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lr4d\" (UniqueName: \"kubernetes.io/projected/398cd612-d4e2-4ac0-b8c4-c632dd73957f-kube-api-access-8lr4d\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:32.896908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896735    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-run-netns\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.896908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896773    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-etc-systemd\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.896908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896802    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf9dp\" (UniqueName: \"kubernetes.io/projected/f9d5138e-b0b1-468a-9170-27760d28c642-kube-api-access-mf9dp\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.896908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896829    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/398cd612-d4e2-4ac0-b8c4-c632dd73957f-cni-binary-copy\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:32.896908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896854    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75a74486-1003-45bb-9c55-1cbf52ab4090-host\") pod \"node-ca-2cwdm\" (UID: \"75a74486-1003-45bb-9c55-1cbf52ab4090\") " pod="openshift-image-registry/node-ca-2cwdm"
Jul 06 19:32:32.896908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896878    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/398cd612-d4e2-4ac0-b8c4-c632dd73957f-os-release\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:32.897671 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896902    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-etc-modprobe-d\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.897671 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896925    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-run-ovn\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.897671 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896950    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-systemd-units\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.897671 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896975    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba-konnectivity-ca\") pod \"konnectivity-agent-cklnx\" (UID: \"b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba\") " pod="kube-system/konnectivity-agent-cklnx"
Jul 06 19:32:32.897671 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.896999    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs\") pod \"network-metrics-daemon-kbl7t\" (UID: \"2bd4ee01-7358-4e6f-9b67-43d41aef4fc3\") " pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:32.897671 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897050    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ea2a94f-253e-4bc7-960f-69e371dbbd6a-hosts-file\") pod \"node-resolver-n9f6w\" (UID: \"5ea2a94f-253e-4bc7-960f-69e371dbbd6a\") " pod="openshift-dns/node-resolver-n9f6w"
Jul 06 19:32:32.897671 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897103    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q87t4\" (UniqueName: \"kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4\") pod \"network-check-target-lvxg5\" (UID: \"2fc30bdf-7515-45f6-9c88-2f976891430f\") " pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:32.897671 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897136    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-etc-openvswitch\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.897671 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897169    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-etc-kubernetes\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.897671 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897196    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/398cd612-d4e2-4ac0-b8c4-c632dd73957f-cnibin\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:32.897671 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897222    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/398cd612-d4e2-4ac0-b8c4-c632dd73957f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:32.897671 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897252    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-run-openvswitch\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.897671 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897276    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-etc-sysconfig\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.897671 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897291    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-host-run-netns\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.897671 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897306    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-multus-daemon-config\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.897671 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897354    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75a74486-1003-45bb-9c55-1cbf52ab4090-serviceca\") pod \"node-ca-2cwdm\" (UID: \"75a74486-1003-45bb-9c55-1cbf52ab4090\") " pod="openshift-image-registry/node-ca-2cwdm"
Jul 06 19:32:32.898244 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897380    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-run-systemd\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.898244 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897416    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-etc-sysctl-conf\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.898244 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897433    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f9d5138e-b0b1-468a-9170-27760d28c642-etc-tuned\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.898244 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897472    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-host-var-lib-kubelet\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.898244 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897518    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-hostroot\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.898244 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897537    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4247m\" (UniqueName: \"kubernetes.io/projected/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-kube-api-access-4247m\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.898244 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897556    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/398cd612-d4e2-4ac0-b8c4-c632dd73957f-system-cni-dir\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:32.898244 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897584    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-var-lib-openvswitch\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.898244 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897607    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-sys\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.898244 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897631    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74p4z\" (UniqueName: \"kubernetes.io/projected/0300b33e-bd11-401e-ac65-312bcd5d72a1-kube-api-access-74p4z\") pod \"iptables-alerter-z5bg8\" (UID: \"0300b33e-bd11-401e-ac65-312bcd5d72a1\") " pod="openshift-network-operator/iptables-alerter-z5bg8"
Jul 06 19:32:32.898244 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897652    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-multus-cni-dir\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.898244 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897678    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/398cd612-d4e2-4ac0-b8c4-c632dd73957f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:32.898244 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897714    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.898244 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897733    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-ovnkube-script-lib\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.898244 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897759    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9d5138e-b0b1-468a-9170-27760d28c642-tmp\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.898244 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897779    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0300b33e-bd11-401e-ac65-312bcd5d72a1-iptables-alerter-script\") pod \"iptables-alerter-z5bg8\" (UID: \"0300b33e-bd11-401e-ac65-312bcd5d72a1\") " pod="openshift-network-operator/iptables-alerter-z5bg8"
Jul 06 19:32:32.898785 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897795    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knvzg\" (UniqueName: \"kubernetes.io/projected/5ea2a94f-253e-4bc7-960f-69e371dbbd6a-kube-api-access-knvzg\") pod \"node-resolver-n9f6w\" (UID: \"5ea2a94f-253e-4bc7-960f-69e371dbbd6a\") " pod="openshift-dns/node-resolver-n9f6w"
Jul 06 19:32:32.898785 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897810    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-os-release\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.898785 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897843    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-host-run-multus-certs\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.898785 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897873    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-cni-bin\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.898785 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897893    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b29855f2-a8a2-49f4-8d0b-7f52c478e852-registration-dir\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:32.898785 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897913    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b29855f2-a8a2-49f4-8d0b-7f52c478e852-device-dir\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:32.898785 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897933    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b29855f2-a8a2-49f4-8d0b-7f52c478e852-etc-selinux\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:32.898785 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897960    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret\") pod \"global-pull-secret-syncer-zgsll\" (UID: \"05300884-0a61-4b61-9637-add8ddfff1c5\") " pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:32.898785 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.897990    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/398cd612-d4e2-4ac0-b8c4-c632dd73957f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:32.898785 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.898020    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-node-log\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.898785 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.898042    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.898785 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.898061    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-etc-sysctl-d\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.998289 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998271    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75a74486-1003-45bb-9c55-1cbf52ab4090-host\") pod \"node-ca-2cwdm\" (UID: \"75a74486-1003-45bb-9c55-1cbf52ab4090\") " pod="openshift-image-registry/node-ca-2cwdm"
Jul 06 19:32:32.998381 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998295    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/398cd612-d4e2-4ac0-b8c4-c632dd73957f-os-release\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:32.998381 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998310    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-etc-modprobe-d\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.998381 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998325    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-run-ovn\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.998381 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998344    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-systemd-units\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.998541 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998409    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-run-ovn\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.998541 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998416    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/398cd612-d4e2-4ac0-b8c4-c632dd73957f-os-release\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:32.998541 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998471    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-etc-modprobe-d\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.998541 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998484    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-systemd-units\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.998541 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998475    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75a74486-1003-45bb-9c55-1cbf52ab4090-host\") pod \"node-ca-2cwdm\" (UID: \"75a74486-1003-45bb-9c55-1cbf52ab4090\") " pod="openshift-image-registry/node-ca-2cwdm"
Jul 06 19:32:32.998541 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998480    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba-konnectivity-ca\") pod \"konnectivity-agent-cklnx\" (UID: \"b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba\") " pod="kube-system/konnectivity-agent-cklnx"
Jul 06 19:32:32.998729 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998549    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs\") pod \"network-metrics-daemon-kbl7t\" (UID: \"2bd4ee01-7358-4e6f-9b67-43d41aef4fc3\") " pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:32.998729 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998573    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ea2a94f-253e-4bc7-960f-69e371dbbd6a-hosts-file\") pod \"node-resolver-n9f6w\" (UID: \"5ea2a94f-253e-4bc7-960f-69e371dbbd6a\") " pod="openshift-dns/node-resolver-n9f6w"
Jul 06 19:32:32.998729 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998624    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ea2a94f-253e-4bc7-960f-69e371dbbd6a-hosts-file\") pod \"node-resolver-n9f6w\" (UID: \"5ea2a94f-253e-4bc7-960f-69e371dbbd6a\") " pod="openshift-dns/node-resolver-n9f6w"
Jul 06 19:32:32.998729 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:32.998653    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 19:32:32.998729 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998691    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q87t4\" (UniqueName: \"kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4\") pod \"network-check-target-lvxg5\" (UID: \"2fc30bdf-7515-45f6-9c88-2f976891430f\") " pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:32.998729 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:32.998725    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs podName:2bd4ee01-7358-4e6f-9b67-43d41aef4fc3 nodeName:}" failed. No retries permitted until 2026-07-06 19:32:33.498695968 +0000 UTC m=+2.039872638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs") pod "network-metrics-daemon-kbl7t" (UID: "2bd4ee01-7358-4e6f-9b67-43d41aef4fc3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 19:32:32.999016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998751    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-etc-openvswitch\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.999016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998773    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-etc-kubernetes\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.999016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998793    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/398cd612-d4e2-4ac0-b8c4-c632dd73957f-cnibin\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:32.999016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998811    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/398cd612-d4e2-4ac0-b8c4-c632dd73957f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:32.999016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998828    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-run-openvswitch\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.999016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998842    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-etc-kubernetes\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.999016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998856    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-etc-sysconfig\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.999016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998871    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-etc-openvswitch\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.999016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998881    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-host-run-netns\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.999016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998896    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/398cd612-d4e2-4ac0-b8c4-c632dd73957f-cnibin\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:32.999016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998915    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-host-run-netns\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.999016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998919    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-etc-sysconfig\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.999016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998929    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-multus-daemon-config\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.999016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998962    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75a74486-1003-45bb-9c55-1cbf52ab4090-serviceca\") pod \"node-ca-2cwdm\" (UID: \"75a74486-1003-45bb-9c55-1cbf52ab4090\") " pod="openshift-image-registry/node-ca-2cwdm"
Jul 06 19:32:32.999016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998969    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-run-openvswitch\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.999016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.998988    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-run-systemd\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.999795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999032    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-run-systemd\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.999795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999034    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba-konnectivity-ca\") pod \"konnectivity-agent-cklnx\" (UID: \"b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba\") " pod="kube-system/konnectivity-agent-cklnx"
Jul 06 19:32:32.999795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999069    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/398cd612-d4e2-4ac0-b8c4-c632dd73957f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:32.999795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-etc-sysctl-conf\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.999795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999130    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f9d5138e-b0b1-468a-9170-27760d28c642-etc-tuned\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.999795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999156    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-host-var-lib-kubelet\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.999795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999181    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-hostroot\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.999795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999206    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-host-var-lib-kubelet\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.999795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4247m\" (UniqueName: \"kubernetes.io/projected/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-kube-api-access-4247m\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.999795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999240    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/398cd612-d4e2-4ac0-b8c4-c632dd73957f-system-cni-dir\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:32.999795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999256    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-etc-sysctl-conf\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.999795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999270    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-var-lib-openvswitch\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.999795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999306    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-hostroot\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:32.999795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999312    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-var-lib-openvswitch\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:32.999795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999305    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/398cd612-d4e2-4ac0-b8c4-c632dd73957f-system-cni-dir\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:32.999795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999328    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-sys\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:32.999795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999369    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74p4z\" (UniqueName: \"kubernetes.io/projected/0300b33e-bd11-401e-ac65-312bcd5d72a1-kube-api-access-74p4z\") pod \"iptables-alerter-z5bg8\" (UID: \"0300b33e-bd11-401e-ac65-312bcd5d72a1\") " pod="openshift-network-operator/iptables-alerter-z5bg8"
Jul 06 19:32:32.999795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999386    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-multus-cni-dir\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.000696 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999411    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75a74486-1003-45bb-9c55-1cbf52ab4090-serviceca\") pod \"node-ca-2cwdm\" (UID: \"75a74486-1003-45bb-9c55-1cbf52ab4090\") " pod="openshift-image-registry/node-ca-2cwdm"
Jul 06 19:32:33.000696 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999436    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-sys\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.000696 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999423    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 06 19:32:33.000696 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999410    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/398cd612-d4e2-4ac0-b8c4-c632dd73957f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:33.000696 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999486    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-multus-cni-dir\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.000696 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999512    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.000696 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999535    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-ovnkube-script-lib\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.000696 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999555    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9d5138e-b0b1-468a-9170-27760d28c642-tmp\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.000696 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999553    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-multus-daemon-config\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.000696 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999571    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0300b33e-bd11-401e-ac65-312bcd5d72a1-iptables-alerter-script\") pod \"iptables-alerter-z5bg8\" (UID: \"0300b33e-bd11-401e-ac65-312bcd5d72a1\") " pod="openshift-network-operator/iptables-alerter-z5bg8"
Jul 06 19:32:33.000696 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999666    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.000696 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999678    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knvzg\" (UniqueName: \"kubernetes.io/projected/5ea2a94f-253e-4bc7-960f-69e371dbbd6a-kube-api-access-knvzg\") pod \"node-resolver-n9f6w\" (UID: \"5ea2a94f-253e-4bc7-960f-69e371dbbd6a\") " pod="openshift-dns/node-resolver-n9f6w"
Jul 06 19:32:33.000696 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999708    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-os-release\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.000696 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-host-run-multus-certs\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.000696 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999760    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-cni-bin\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.000696 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999782    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b29855f2-a8a2-49f4-8d0b-7f52c478e852-registration-dir\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:33.000696 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999806    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b29855f2-a8a2-49f4-8d0b-7f52c478e852-device-dir\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:33.001552 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999830    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b29855f2-a8a2-49f4-8d0b-7f52c478e852-etc-selinux\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:33.001552 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999856    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret\") pod \"global-pull-secret-syncer-zgsll\" (UID: \"05300884-0a61-4b61-9637-add8ddfff1c5\") " pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:33.001552 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999885    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/398cd612-d4e2-4ac0-b8c4-c632dd73957f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:33.001552 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999910    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-node-log\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.001552 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999932    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.001552 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999943    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b29855f2-a8a2-49f4-8d0b-7f52c478e852-registration-dir\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:33.001552 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999960    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-etc-sysctl-d\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.001552 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:32.999983    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-run\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.001552 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000009    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-var-lib-kubelet\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.001552 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000034    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-host\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.001552 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000047    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-host-run-multus-certs\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.001552 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000059    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b29855f2-a8a2-49f4-8d0b-7f52c478e852-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:33.001552 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000090    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-cni-bin\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.001552 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm9gf\" (UniqueName: \"kubernetes.io/projected/75a74486-1003-45bb-9c55-1cbf52ab4090-kube-api-access-gm9gf\") pod \"node-ca-2cwdm\" (UID: \"75a74486-1003-45bb-9c55-1cbf52ab4090\") " pod="openshift-image-registry/node-ca-2cwdm"
Jul 06 19:32:33.001552 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000118    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/398cd612-d4e2-4ac0-b8c4-c632dd73957f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:33.001552 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000132    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-kubelet\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.001552 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000144    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-ovnkube-script-lib\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.002385 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000168    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-ovn-node-metrics-cert\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.002385 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000192    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.002385 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000193    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-host-var-lib-cni-bin\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.002385 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000218    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-host-var-lib-cni-multus\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.002385 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000224    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-host-var-lib-cni-bin\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.002385 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000235    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-multus-conf-dir\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.002385 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000251    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-etc-kubernetes\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.002385 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000256    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-var-lib-kubelet\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.002385 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000267    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-multus-socket-dir-parent\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.002385 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000284    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xjd\" (UniqueName: \"kubernetes.io/projected/b29855f2-a8a2-49f4-8d0b-7f52c478e852-kube-api-access-97xjd\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:33.002385 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000300    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-slash\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.002385 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000316    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-ovnkube-config\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.002385 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000331    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-lib-modules\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.002385 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000345    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-cnibin\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.002385 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000360    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-cni-binary-copy\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.002385 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000384    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b29855f2-a8a2-49f4-8d0b-7f52c478e852-socket-dir\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:33.002385 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000400    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-etc-sysctl-d\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.002933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000410    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-log-socket\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.002933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000438    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-cni-netd\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.002933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000451    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-run\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.002933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000463    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-env-overrides\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.002933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000490    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba-agent-certs\") pod \"konnectivity-agent-cklnx\" (UID: \"b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba\") " pod="kube-system/konnectivity-agent-cklnx"
Jul 06 19:32:33.002933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000518    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05300884-0a61-4b61-9637-add8ddfff1c5-kubelet-config\") pod \"global-pull-secret-syncer-zgsll\" (UID: \"05300884-0a61-4b61-9637-add8ddfff1c5\") " pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:33.002933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000543    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztm2q\" (UniqueName: \"kubernetes.io/projected/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-kube-api-access-ztm2q\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.002933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000591    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn4fv\" (UniqueName: \"kubernetes.io/projected/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-kube-api-access-nn4fv\") pod \"network-metrics-daemon-kbl7t\" (UID: \"2bd4ee01-7358-4e6f-9b67-43d41aef4fc3\") " pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:33.002933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000601    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-host\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.002933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000011    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-os-release\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.002933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000622    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0300b33e-bd11-401e-ac65-312bcd5d72a1-host-slash\") pod \"iptables-alerter-z5bg8\" (UID: \"0300b33e-bd11-401e-ac65-312bcd5d72a1\") " pod="openshift-network-operator/iptables-alerter-z5bg8"
Jul 06 19:32:33.002933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000169    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-kubelet\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.002933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000731    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-host-run-k8s-cni-cncf-io\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.002933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000743    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0300b33e-bd11-401e-ac65-312bcd5d72a1-iptables-alerter-script\") pod \"iptables-alerter-z5bg8\" (UID: \"0300b33e-bd11-401e-ac65-312bcd5d72a1\") " pod="openshift-network-operator/iptables-alerter-z5bg8"
Jul 06 19:32:33.002933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000767    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-host-var-lib-cni-multus\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.002933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000760    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05300884-0a61-4b61-9637-add8ddfff1c5-dbus\") pod \"global-pull-secret-syncer-zgsll\" (UID: \"05300884-0a61-4b61-9637-add8ddfff1c5\") " pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:33.002933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000806    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b29855f2-a8a2-49f4-8d0b-7f52c478e852-device-dir\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:33.002933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000809    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea2a94f-253e-4bc7-960f-69e371dbbd6a-tmp-dir\") pod \"node-resolver-n9f6w\" (UID: \"5ea2a94f-253e-4bc7-960f-69e371dbbd6a\") " pod="openshift-dns/node-resolver-n9f6w"
Jul 06 19:32:33.003658 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000825    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05300884-0a61-4b61-9637-add8ddfff1c5-dbus\") pod \"global-pull-secret-syncer-zgsll\" (UID: \"05300884-0a61-4b61-9637-add8ddfff1c5\") " pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:33.003658 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000836    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-system-cni-dir\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.003658 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000842    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-multus-conf-dir\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.003658 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000864    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b29855f2-a8a2-49f4-8d0b-7f52c478e852-sys-fs\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:33.003658 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000908    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b29855f2-a8a2-49f4-8d0b-7f52c478e852-etc-selinux\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:33.003658 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000917    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lr4d\" (UniqueName: \"kubernetes.io/projected/398cd612-d4e2-4ac0-b8c4-c632dd73957f-kube-api-access-8lr4d\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:33.003658 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000951    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-run-netns\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.003658 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000961    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-multus-socket-dir-parent\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.003658 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000976    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-etc-systemd\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.003658 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001004    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf9dp\" (UniqueName: \"kubernetes.io/projected/f9d5138e-b0b1-468a-9170-27760d28c642-kube-api-access-mf9dp\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.003658 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001034    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/398cd612-d4e2-4ac0-b8c4-c632dd73957f-cni-binary-copy\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:33.003658 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001052    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/398cd612-d4e2-4ac0-b8c4-c632dd73957f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:33.003658 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.000733    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b29855f2-a8a2-49f4-8d0b-7f52c478e852-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:33.003658 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001055    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-lib-modules\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.003658 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001117    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-run-netns\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.003658 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001135    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-etc-systemd\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.003658 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001161    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05300884-0a61-4b61-9637-add8ddfff1c5-kubelet-config\") pod \"global-pull-secret-syncer-zgsll\" (UID: \"05300884-0a61-4b61-9637-add8ddfff1c5\") " pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:33.004357 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001299    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-node-log\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.004357 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001475    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea2a94f-253e-4bc7-960f-69e371dbbd6a-tmp-dir\") pod \"node-resolver-n9f6w\" (UID: \"5ea2a94f-253e-4bc7-960f-69e371dbbd6a\") " pod="openshift-dns/node-resolver-n9f6w"
Jul 06 19:32:33.004357 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001486    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b29855f2-a8a2-49f4-8d0b-7f52c478e852-socket-dir\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:33.004357 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001533    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-log-socket\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.004357 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001578    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-system-cni-dir\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.004357 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001587    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-cni-netd\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.004357 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001658    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9d5138e-b0b1-468a-9170-27760d28c642-etc-kubernetes\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.004357 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001697    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-host-slash\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.004357 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001741    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b29855f2-a8a2-49f4-8d0b-7f52c478e852-sys-fs\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:33.004357 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001759    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-cni-binary-copy\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.004357 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001819    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/398cd612-d4e2-4ac0-b8c4-c632dd73957f-cni-binary-copy\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:33.004357 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001854    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0300b33e-bd11-401e-ac65-312bcd5d72a1-host-slash\") pod \"iptables-alerter-z5bg8\" (UID: \"0300b33e-bd11-401e-ac65-312bcd5d72a1\") " pod="openshift-network-operator/iptables-alerter-z5bg8"
Jul 06 19:32:33.004357 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001903    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-host-run-k8s-cni-cncf-io\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.004357 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.002188    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-env-overrides\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.004357 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.002207    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-ovnkube-config\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.004357 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.001800    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-cnibin\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.004357 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:33.002280    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 19:32:33.004357 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:33.002334    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret podName:05300884-0a61-4b61-9637-add8ddfff1c5 nodeName:}" failed. No retries permitted until 2026-07-06 19:32:33.502318053 +0000 UTC m=+2.043494719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret") pod "global-pull-secret-syncer-zgsll" (UID: "05300884-0a61-4b61-9637-add8ddfff1c5") : object "kube-system"/"original-pull-secret" not registered
Jul 06 19:32:33.004943 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.003349    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9d5138e-b0b1-468a-9170-27760d28c642-tmp\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.004943 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.003354    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f9d5138e-b0b1-468a-9170-27760d28c642-etc-tuned\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.004943 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.003474    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-ovn-node-metrics-cert\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.004943 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.004318    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba-agent-certs\") pod \"konnectivity-agent-cklnx\" (UID: \"b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba\") " pod="kube-system/konnectivity-agent-cklnx"
Jul 06 19:32:33.009188 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:33.009170    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 19:32:33.009264 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:33.009193    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 19:32:33.009264 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:33.009206    2562 projected.go:196] Error preparing data for projected volume kube-api-access-q87t4 for pod openshift-network-diagnostics/network-check-target-lvxg5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 19:32:33.009264 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:33.009256    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4 podName:2fc30bdf-7515-45f6-9c88-2f976891430f nodeName:}" failed. No retries permitted until 2026-07-06 19:32:33.509245014 +0000 UTC m=+2.050421674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q87t4" (UniqueName: "kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4") pod "network-check-target-lvxg5" (UID: "2fc30bdf-7515-45f6-9c88-2f976891430f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 19:32:33.011705 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.011682    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4247m\" (UniqueName: \"kubernetes.io/projected/1a262cc1-2cd2-40f0-ba32-fc612faf3c4a-kube-api-access-4247m\") pod \"multus-tkgjn\" (UID: \"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a\") " pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.014958 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.014899    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztm2q\" (UniqueName: \"kubernetes.io/projected/d2b1ab09-9428-4ccf-bcf1-fe21564d41f4-kube-api-access-ztm2q\") pod \"ovnkube-node-wt558\" (UID: \"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.017097 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.016429    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knvzg\" (UniqueName: \"kubernetes.io/projected/5ea2a94f-253e-4bc7-960f-69e371dbbd6a-kube-api-access-knvzg\") pod \"node-resolver-n9f6w\" (UID: \"5ea2a94f-253e-4bc7-960f-69e371dbbd6a\") " pod="openshift-dns/node-resolver-n9f6w"
Jul 06 19:32:33.017194 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.017116    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lr4d\" (UniqueName: \"kubernetes.io/projected/398cd612-d4e2-4ac0-b8c4-c632dd73957f-kube-api-access-8lr4d\") pod \"multus-additional-cni-plugins-5rwdf\" (UID: \"398cd612-d4e2-4ac0-b8c4-c632dd73957f\") " pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:33.018145 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.018125    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm9gf\" (UniqueName: \"kubernetes.io/projected/75a74486-1003-45bb-9c55-1cbf52ab4090-kube-api-access-gm9gf\") pod \"node-ca-2cwdm\" (UID: \"75a74486-1003-45bb-9c55-1cbf52ab4090\") " pod="openshift-image-registry/node-ca-2cwdm"
Jul 06 19:32:33.018251 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.018169    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74p4z\" (UniqueName: \"kubernetes.io/projected/0300b33e-bd11-401e-ac65-312bcd5d72a1-kube-api-access-74p4z\") pod \"iptables-alerter-z5bg8\" (UID: \"0300b33e-bd11-401e-ac65-312bcd5d72a1\") " pod="openshift-network-operator/iptables-alerter-z5bg8"
Jul 06 19:32:33.018311 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.018288    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn4fv\" (UniqueName: \"kubernetes.io/projected/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-kube-api-access-nn4fv\") pod \"network-metrics-daemon-kbl7t\" (UID: \"2bd4ee01-7358-4e6f-9b67-43d41aef4fc3\") " pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:33.019009 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.018991    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97xjd\" (UniqueName: \"kubernetes.io/projected/b29855f2-a8a2-49f4-8d0b-7f52c478e852-kube-api-access-97xjd\") pod \"aws-ebs-csi-driver-node-l7fpt\" (UID: \"b29855f2-a8a2-49f4-8d0b-7f52c478e852\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:33.019280 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.019258    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf9dp\" (UniqueName: \"kubernetes.io/projected/f9d5138e-b0b1-468a-9170-27760d28c642-kube-api-access-mf9dp\") pod \"tuned-dfspr\" (UID: \"f9d5138e-b0b1-468a-9170-27760d28c642\") " pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.127695 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.127638    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tkgjn"
Jul 06 19:32:33.144727 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.144709    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:32:33.166863 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.166839    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cklnx"
Jul 06 19:32:33.169149 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.169132    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt"
Jul 06 19:32:33.197657 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.197639    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dfspr"
Jul 06 19:32:33.211519 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.211500    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z5bg8"
Jul 06 19:32:33.211963 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:33.211938    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ecf9c2_c7ab_440a_b3b3_3db6bee193ba.slice/crio-5079909cc086981a55a1798a2b202cdd91cc2c72492e0ec05f76a7871aa69935 WatchSource:0}: Error finding container 5079909cc086981a55a1798a2b202cdd91cc2c72492e0ec05f76a7871aa69935: Status 404 returned error can't find the container with id 5079909cc086981a55a1798a2b202cdd91cc2c72492e0ec05f76a7871aa69935
Jul 06 19:32:33.212712 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:33.212690    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc01bcdac46728d38f70a288423f3cc9.slice/crio-ed372e35201426b6cb4ab85900c17681cff342918878b796399c2a96e6699690 WatchSource:0}: Error finding container ed372e35201426b6cb4ab85900c17681cff342918878b796399c2a96e6699690: Status 404 returned error can't find the container with id ed372e35201426b6cb4ab85900c17681cff342918878b796399c2a96e6699690
Jul 06 19:32:33.217706 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.217684    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 19:32:33.218933 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:33.218916    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d5138e_b0b1_468a_9170_27760d28c642.slice/crio-62fc702a6bb7224f2ef133c00e96b58bc50daf2d428571819804b5d44065515a WatchSource:0}: Error finding container 62fc702a6bb7224f2ef133c00e96b58bc50daf2d428571819804b5d44065515a: Status 404 returned error can't find the container with id 62fc702a6bb7224f2ef133c00e96b58bc50daf2d428571819804b5d44065515a
Jul 06 19:32:33.221528 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:33.221477    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd31cf43040b67356073a3b42229e6bd.slice/crio-54dfc9a590faf5ed5a464bccfb1c48a3937a9c0ac9855d79e688c2ed13fb7fa7 WatchSource:0}: Error finding container 54dfc9a590faf5ed5a464bccfb1c48a3937a9c0ac9855d79e688c2ed13fb7fa7: Status 404 returned error can't find the container with id 54dfc9a590faf5ed5a464bccfb1c48a3937a9c0ac9855d79e688c2ed13fb7fa7
Jul 06 19:32:33.225029 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:33.225010    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0300b33e_bd11_401e_ac65_312bcd5d72a1.slice/crio-e5b3f3401edd3d9ee7a4887395140675c11976f0ed470cbef1a5dccd558c8ef6 WatchSource:0}: Error finding container e5b3f3401edd3d9ee7a4887395140675c11976f0ed470cbef1a5dccd558c8ef6: Status 404 returned error can't find the container with id e5b3f3401edd3d9ee7a4887395140675c11976f0ed470cbef1a5dccd558c8ef6
Jul 06 19:32:33.226321 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.226303    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n9f6w"
Jul 06 19:32:33.236007 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:33.235989    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea2a94f_253e_4bc7_960f_69e371dbbd6a.slice/crio-c0a4ec4936bb5bcc5d2a87cbae108b5e77979be6cf76fc47582a7cb6cade777b WatchSource:0}: Error finding container c0a4ec4936bb5bcc5d2a87cbae108b5e77979be6cf76fc47582a7cb6cade777b: Status 404 returned error can't find the container with id c0a4ec4936bb5bcc5d2a87cbae108b5e77979be6cf76fc47582a7cb6cade777b
Jul 06 19:32:33.244570 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.244544    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2cwdm"
Jul 06 19:32:33.249587 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.249556    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5rwdf"
Jul 06 19:32:33.252377 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:33.252284    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a74486_1003_45bb_9c55_1cbf52ab4090.slice/crio-726a8d256473aafef2a82745e1c62a2c25627047430e23a793a55163440a2d66 WatchSource:0}: Error finding container 726a8d256473aafef2a82745e1c62a2c25627047430e23a793a55163440a2d66: Status 404 returned error can't find the container with id 726a8d256473aafef2a82745e1c62a2c25627047430e23a793a55163440a2d66
Jul 06 19:32:33.259173 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:33.259155    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398cd612_d4e2_4ac0_b8c4_c632dd73957f.slice/crio-ec1c8357a86e3868324b4708aa35ccfa9f605d316de0ef1a5038b8fbdf5bffca WatchSource:0}: Error finding container ec1c8357a86e3868324b4708aa35ccfa9f605d316de0ef1a5038b8fbdf5bffca: Status 404 returned error can't find the container with id ec1c8357a86e3868324b4708aa35ccfa9f605d316de0ef1a5038b8fbdf5bffca
Jul 06 19:32:33.506022 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.505955    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs\") pod \"network-metrics-daemon-kbl7t\" (UID: \"2bd4ee01-7358-4e6f-9b67-43d41aef4fc3\") " pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:33.506022 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.506015    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret\") pod \"global-pull-secret-syncer-zgsll\" (UID: \"05300884-0a61-4b61-9637-add8ddfff1c5\") " pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:33.506260 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:33.506091    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 19:32:33.506260 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:33.506130    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 19:32:33.506260 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:33.506145    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs podName:2bd4ee01-7358-4e6f-9b67-43d41aef4fc3 nodeName:}" failed. No retries permitted until 2026-07-06 19:32:34.50613047 +0000 UTC m=+3.047307125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs") pod "network-metrics-daemon-kbl7t" (UID: "2bd4ee01-7358-4e6f-9b67-43d41aef4fc3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 19:32:33.506260 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:33.506184    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret podName:05300884-0a61-4b61-9637-add8ddfff1c5 nodeName:}" failed. No retries permitted until 2026-07-06 19:32:34.506168189 +0000 UTC m=+3.047344860 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret") pod "global-pull-secret-syncer-zgsll" (UID: "05300884-0a61-4b61-9637-add8ddfff1c5") : object "kube-system"/"original-pull-secret" not registered
Jul 06 19:32:33.530267 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:33.530247    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29855f2_a8a2_49f4_8d0b_7f52c478e852.slice/crio-ec3b078d6dd888d9639df10a72fb20b28f1f555cf3adc405eccc06f7ec607252 WatchSource:0}: Error finding container ec3b078d6dd888d9639df10a72fb20b28f1f555cf3adc405eccc06f7ec607252: Status 404 returned error can't find the container with id ec3b078d6dd888d9639df10a72fb20b28f1f555cf3adc405eccc06f7ec607252
Jul 06 19:32:33.566213 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.566188    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 19:32:33.581855 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:33.581836    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b1ab09_9428_4ccf_bcf1_fe21564d41f4.slice/crio-2055c900a6c4d71f76d6caba8de2f409e58b825bd756ed832231e26d4bd7e621 WatchSource:0}: Error finding container 2055c900a6c4d71f76d6caba8de2f409e58b825bd756ed832231e26d4bd7e621: Status 404 returned error can't find the container with id 2055c900a6c4d71f76d6caba8de2f409e58b825bd756ed832231e26d4bd7e621
Jul 06 19:32:33.607341 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.607318    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q87t4\" (UniqueName: \"kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4\") pod \"network-check-target-lvxg5\" (UID: \"2fc30bdf-7515-45f6-9c88-2f976891430f\") " pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:33.607437 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:33.607424    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 19:32:33.607479 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:33.607472    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 19:32:33.607529 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:33.607484    2562 projected.go:196] Error preparing data for projected volume kube-api-access-q87t4 for pod openshift-network-diagnostics/network-check-target-lvxg5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 19:32:33.607566 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:33.607547    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4 podName:2fc30bdf-7515-45f6-9c88-2f976891430f nodeName:}" failed. No retries permitted until 2026-07-06 19:32:34.607531952 +0000 UTC m=+3.148708611 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q87t4" (UniqueName: "kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4") pod "network-check-target-lvxg5" (UID: "2fc30bdf-7515-45f6-9c88-2f976891430f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 19:32:33.622407 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:32:33.622388    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a262cc1_2cd2_40f0_ba32_fc612faf3c4a.slice/crio-2212c70b7f10338741077e7adb9cf2c2bc5b3600c71503dfea1fe31caa57394b WatchSource:0}: Error finding container 2212c70b7f10338741077e7adb9cf2c2bc5b3600c71503dfea1fe31caa57394b: Status 404 returned error can't find the container with id 2212c70b7f10338741077e7adb9cf2c2bc5b3600c71503dfea1fe31caa57394b
Jul 06 19:32:33.854504 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.854427    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:27:33 +0000 UTC" deadline="2028-03-12 09:33:12.761999662 +0000 UTC"
Jul 06 19:32:33.854504 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.854450    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14750h0m38.907552208s"
Jul 06 19:32:33.893704 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.893646    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt" event={"ID":"b29855f2-a8a2-49f4-8d0b-7f52c478e852","Type":"ContainerStarted","Data":"ec3b078d6dd888d9639df10a72fb20b28f1f555cf3adc405eccc06f7ec607252"}
Jul 06 19:32:33.895174 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.895146    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rwdf" event={"ID":"398cd612-d4e2-4ac0-b8c4-c632dd73957f","Type":"ContainerStarted","Data":"ec1c8357a86e3868324b4708aa35ccfa9f605d316de0ef1a5038b8fbdf5bffca"}
Jul 06 19:32:33.896860 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.896806    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2cwdm" event={"ID":"75a74486-1003-45bb-9c55-1cbf52ab4090","Type":"ContainerStarted","Data":"726a8d256473aafef2a82745e1c62a2c25627047430e23a793a55163440a2d66"}
Jul 06 19:32:33.898718 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.898695    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dfspr" event={"ID":"f9d5138e-b0b1-468a-9170-27760d28c642","Type":"ContainerStarted","Data":"62fc702a6bb7224f2ef133c00e96b58bc50daf2d428571819804b5d44065515a"}
Jul 06 19:32:33.900113 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.900059    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n9f6w" event={"ID":"5ea2a94f-253e-4bc7-960f-69e371dbbd6a","Type":"ContainerStarted","Data":"c0a4ec4936bb5bcc5d2a87cbae108b5e77979be6cf76fc47582a7cb6cade777b"}
Jul 06 19:32:33.901440 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.901408    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z5bg8" event={"ID":"0300b33e-bd11-401e-ac65-312bcd5d72a1","Type":"ContainerStarted","Data":"e5b3f3401edd3d9ee7a4887395140675c11976f0ed470cbef1a5dccd558c8ef6"}
Jul 06 19:32:33.903049 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.903022    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal" event={"ID":"cd31cf43040b67356073a3b42229e6bd","Type":"ContainerStarted","Data":"54dfc9a590faf5ed5a464bccfb1c48a3937a9c0ac9855d79e688c2ed13fb7fa7"}
Jul 06 19:32:33.904591 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.904526    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-200.ec2.internal" event={"ID":"bc01bcdac46728d38f70a288423f3cc9","Type":"ContainerStarted","Data":"ed372e35201426b6cb4ab85900c17681cff342918878b796399c2a96e6699690"}
Jul 06 19:32:33.905934 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.905901    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cklnx" event={"ID":"b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba","Type":"ContainerStarted","Data":"5079909cc086981a55a1798a2b202cdd91cc2c72492e0ec05f76a7871aa69935"}
Jul 06 19:32:33.907973 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.907947    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tkgjn" event={"ID":"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a","Type":"ContainerStarted","Data":"2212c70b7f10338741077e7adb9cf2c2bc5b3600c71503dfea1fe31caa57394b"}
Jul 06 19:32:33.909865 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:33.909841    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wt558" event={"ID":"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4","Type":"ContainerStarted","Data":"2055c900a6c4d71f76d6caba8de2f409e58b825bd756ed832231e26d4bd7e621"}
Jul 06 19:32:34.516314 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:34.516275    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret\") pod \"global-pull-secret-syncer-zgsll\" (UID: \"05300884-0a61-4b61-9637-add8ddfff1c5\") " pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:34.516514 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:34.516353    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs\") pod \"network-metrics-daemon-kbl7t\" (UID: \"2bd4ee01-7358-4e6f-9b67-43d41aef4fc3\") " pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:34.516514 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:34.516477    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 19:32:34.516678 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:34.516563    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs podName:2bd4ee01-7358-4e6f-9b67-43d41aef4fc3 nodeName:}" failed. No retries permitted until 2026-07-06 19:32:36.516524287 +0000 UTC m=+5.057700968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs") pod "network-metrics-daemon-kbl7t" (UID: "2bd4ee01-7358-4e6f-9b67-43d41aef4fc3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 19:32:34.516678 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:34.516637    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 19:32:34.516873 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:34.516681    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret podName:05300884-0a61-4b61-9637-add8ddfff1c5 nodeName:}" failed. No retries permitted until 2026-07-06 19:32:36.51666726 +0000 UTC m=+5.057843937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret") pod "global-pull-secret-syncer-zgsll" (UID: "05300884-0a61-4b61-9637-add8ddfff1c5") : object "kube-system"/"original-pull-secret" not registered
Jul 06 19:32:34.617664 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:34.617627    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q87t4\" (UniqueName: \"kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4\") pod \"network-check-target-lvxg5\" (UID: \"2fc30bdf-7515-45f6-9c88-2f976891430f\") " pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:34.617852 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:34.617831    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 19:32:34.617852 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:34.617851    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 19:32:34.617972 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:34.617865    2562 projected.go:196] Error preparing data for projected volume kube-api-access-q87t4 for pod openshift-network-diagnostics/network-check-target-lvxg5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 19:32:34.617972 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:34.617927    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4 podName:2fc30bdf-7515-45f6-9c88-2f976891430f nodeName:}" failed. No retries permitted until 2026-07-06 19:32:36.61790843 +0000 UTC m=+5.159085092 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q87t4" (UniqueName: "kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4") pod "network-check-target-lvxg5" (UID: "2fc30bdf-7515-45f6-9c88-2f976891430f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 19:32:34.854860 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:34.854742    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:27:33 +0000 UTC" deadline="2028-02-25 23:35:11.350518115 +0000 UTC"
Jul 06 19:32:34.854860 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:34.854780    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14380h2m36.495743672s"
Jul 06 19:32:34.888639 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:34.888612    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:34.888788 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:34.888738    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgsll" podUID="05300884-0a61-4b61-9637-add8ddfff1c5"
Jul 06 19:32:34.889158 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:34.889137    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:34.889259 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:34.889238    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvxg5" podUID="2fc30bdf-7515-45f6-9c88-2f976891430f"
Jul 06 19:32:34.889326 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:34.889310    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:34.889428 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:34.889396    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbl7t" podUID="2bd4ee01-7358-4e6f-9b67-43d41aef4fc3"
Jul 06 19:32:35.624245 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:35.624207    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 19:32:36.535137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:36.534102    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret\") pod \"global-pull-secret-syncer-zgsll\" (UID: \"05300884-0a61-4b61-9637-add8ddfff1c5\") " pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:36.535137 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:36.534175    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs\") pod \"network-metrics-daemon-kbl7t\" (UID: \"2bd4ee01-7358-4e6f-9b67-43d41aef4fc3\") " pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:36.535137 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:36.534319    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 19:32:36.535137 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:36.534380    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs podName:2bd4ee01-7358-4e6f-9b67-43d41aef4fc3 nodeName:}" failed. No retries permitted until 2026-07-06 19:32:40.534360523 +0000 UTC m=+9.075537183 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs") pod "network-metrics-daemon-kbl7t" (UID: "2bd4ee01-7358-4e6f-9b67-43d41aef4fc3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 19:32:36.535137 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:36.535023    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 19:32:36.535137 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:36.535078    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret podName:05300884-0a61-4b61-9637-add8ddfff1c5 nodeName:}" failed. No retries permitted until 2026-07-06 19:32:40.535061009 +0000 UTC m=+9.076237671 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret") pod "global-pull-secret-syncer-zgsll" (UID: "05300884-0a61-4b61-9637-add8ddfff1c5") : object "kube-system"/"original-pull-secret" not registered
Jul 06 19:32:36.635061 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:36.635022    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q87t4\" (UniqueName: \"kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4\") pod \"network-check-target-lvxg5\" (UID: \"2fc30bdf-7515-45f6-9c88-2f976891430f\") " pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:36.635357 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:36.635338    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 19:32:36.635463 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:36.635364    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 19:32:36.635463 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:36.635378    2562 projected.go:196] Error preparing data for projected volume kube-api-access-q87t4 for pod openshift-network-diagnostics/network-check-target-lvxg5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 19:32:36.635463 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:36.635437    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4 podName:2fc30bdf-7515-45f6-9c88-2f976891430f nodeName:}" failed. No retries permitted until 2026-07-06 19:32:40.635419608 +0000 UTC m=+9.176596273 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q87t4" (UniqueName: "kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4") pod "network-check-target-lvxg5" (UID: "2fc30bdf-7515-45f6-9c88-2f976891430f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 19:32:36.890184 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:36.890105    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:36.890349 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:36.890222    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvxg5" podUID="2fc30bdf-7515-45f6-9c88-2f976891430f"
Jul 06 19:32:36.890681 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:36.890658    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:36.890808 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:36.890773    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbl7t" podUID="2bd4ee01-7358-4e6f-9b67-43d41aef4fc3"
Jul 06 19:32:36.890880 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:36.890840    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:36.890940 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:36.890916    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgsll" podUID="05300884-0a61-4b61-9637-add8ddfff1c5"
Jul 06 19:32:38.890360 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:38.889611    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:38.890360 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:38.889751    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgsll" podUID="05300884-0a61-4b61-9637-add8ddfff1c5"
Jul 06 19:32:38.890360 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:38.890125    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:38.890360 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:38.890228    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvxg5" podUID="2fc30bdf-7515-45f6-9c88-2f976891430f"
Jul 06 19:32:38.890360 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:38.890281    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:38.890360 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:38.890366    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbl7t" podUID="2bd4ee01-7358-4e6f-9b67-43d41aef4fc3"
Jul 06 19:32:40.568401 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:40.568354    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs\") pod \"network-metrics-daemon-kbl7t\" (UID: \"2bd4ee01-7358-4e6f-9b67-43d41aef4fc3\") " pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:40.568876 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:40.568438    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret\") pod \"global-pull-secret-syncer-zgsll\" (UID: \"05300884-0a61-4b61-9637-add8ddfff1c5\") " pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:40.568876 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:40.568517    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 19:32:40.568876 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:40.568589    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 19:32:40.568876 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:40.568594    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs podName:2bd4ee01-7358-4e6f-9b67-43d41aef4fc3 nodeName:}" failed. No retries permitted until 2026-07-06 19:32:48.568572201 +0000 UTC m=+17.109748860 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs") pod "network-metrics-daemon-kbl7t" (UID: "2bd4ee01-7358-4e6f-9b67-43d41aef4fc3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 19:32:40.568876 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:40.568655    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret podName:05300884-0a61-4b61-9637-add8ddfff1c5 nodeName:}" failed. No retries permitted until 2026-07-06 19:32:48.568638325 +0000 UTC m=+17.109814980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret") pod "global-pull-secret-syncer-zgsll" (UID: "05300884-0a61-4b61-9637-add8ddfff1c5") : object "kube-system"/"original-pull-secret" not registered
Jul 06 19:32:40.668841 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:40.668732    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q87t4\" (UniqueName: \"kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4\") pod \"network-check-target-lvxg5\" (UID: \"2fc30bdf-7515-45f6-9c88-2f976891430f\") " pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:40.669013 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:40.668898    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 19:32:40.669013 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:40.668919    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 19:32:40.669013 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:40.668930    2562 projected.go:196] Error preparing data for projected volume kube-api-access-q87t4 for pod openshift-network-diagnostics/network-check-target-lvxg5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 19:32:40.669013 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:40.668993    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4 podName:2fc30bdf-7515-45f6-9c88-2f976891430f nodeName:}" failed. No retries permitted until 2026-07-06 19:32:48.668975019 +0000 UTC m=+17.210151676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q87t4" (UniqueName: "kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4") pod "network-check-target-lvxg5" (UID: "2fc30bdf-7515-45f6-9c88-2f976891430f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 19:32:40.889415 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:40.889196    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:40.889415 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:40.889340    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbl7t" podUID="2bd4ee01-7358-4e6f-9b67-43d41aef4fc3"
Jul 06 19:32:40.889689 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:40.889620    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:40.889749 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:40.889724    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgsll" podUID="05300884-0a61-4b61-9637-add8ddfff1c5"
Jul 06 19:32:40.889805 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:40.889768    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:40.889867 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:40.889832    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvxg5" podUID="2fc30bdf-7515-45f6-9c88-2f976891430f"
Jul 06 19:32:42.889264 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:42.889190    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:42.889264 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:42.889229    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:42.889813 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:42.889190    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:42.889813 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:42.889324    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbl7t" podUID="2bd4ee01-7358-4e6f-9b67-43d41aef4fc3"
Jul 06 19:32:42.889813 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:42.889380    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgsll" podUID="05300884-0a61-4b61-9637-add8ddfff1c5"
Jul 06 19:32:42.889813 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:42.889439    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvxg5" podUID="2fc30bdf-7515-45f6-9c88-2f976891430f"
Jul 06 19:32:44.889388 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:44.889358    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:44.889874 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:44.889479    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:44.889874 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:44.889480    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvxg5" podUID="2fc30bdf-7515-45f6-9c88-2f976891430f"
Jul 06 19:32:44.889874 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:44.889501    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:44.889874 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:44.889617    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbl7t" podUID="2bd4ee01-7358-4e6f-9b67-43d41aef4fc3"
Jul 06 19:32:44.889874 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:44.889697    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgsll" podUID="05300884-0a61-4b61-9637-add8ddfff1c5"
Jul 06 19:32:45.950337 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:45.950298    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-200.ec2.internal" event={"ID":"bc01bcdac46728d38f70a288423f3cc9","Type":"ContainerStarted","Data":"4a7fe8fd75803fe1963b2a1f87a9a9bc95ef1f95a22850f07ffaded6b37fd1b6"}
Jul 06 19:32:45.967617 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:45.967577    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-200.ec2.internal" podStartSLOduration=12.967563409 podStartE2EDuration="12.967563409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:45.967109609 +0000 UTC m=+14.508286289" watchObservedRunningTime="2026-07-06 19:32:45.967563409 +0000 UTC m=+14.508740084"
Jul 06 19:32:46.889041 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:46.889014    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:46.889604 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:46.889116    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:46.889604 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:46.889128    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvxg5" podUID="2fc30bdf-7515-45f6-9c88-2f976891430f"
Jul 06 19:32:46.889604 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:46.889144    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:46.889604 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:46.889214    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbl7t" podUID="2bd4ee01-7358-4e6f-9b67-43d41aef4fc3"
Jul 06 19:32:46.889604 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:46.889271    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgsll" podUID="05300884-0a61-4b61-9637-add8ddfff1c5"
Jul 06 19:32:46.953622 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:46.953593    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dfspr" event={"ID":"f9d5138e-b0b1-468a-9170-27760d28c642","Type":"ContainerStarted","Data":"6a8ae4dd99eeb04c0de07ee416057259d98c4dbbb8831459316dee596fa0564e"}
Jul 06 19:32:47.957233 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:47.956960    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n9f6w" event={"ID":"5ea2a94f-253e-4bc7-960f-69e371dbbd6a","Type":"ContainerStarted","Data":"de8630a8eb606bafa402af25cc428a233221ea182915f86dd3f8bf758b553d7a"}
Jul 06 19:32:47.958616 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:47.958591    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z5bg8" event={"ID":"0300b33e-bd11-401e-ac65-312bcd5d72a1","Type":"ContainerStarted","Data":"09a908b7411e20ce11dad6a3b26f79abf9d3d6d548ec8a8e8c029de5bb30f6d6"}
Jul 06 19:32:47.960220 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:47.960193    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal" event={"ID":"cd31cf43040b67356073a3b42229e6bd","Type":"ContainerStarted","Data":"de81d4078ca2bd8248b85064d236b59de2f6bd16438add05b4f93ba62e459bb1"}
Jul 06 19:32:47.961873 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:47.961837    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cklnx" event={"ID":"b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba","Type":"ContainerStarted","Data":"df918b98f7e4e61fd55d7534a045ce22d0bdbe3a635a35aefe217dd0bb63e262"}
Jul 06 19:32:47.963454 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:47.963428    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt" event={"ID":"b29855f2-a8a2-49f4-8d0b-7f52c478e852","Type":"ContainerStarted","Data":"7cf832254bf51473c578b6ed766f8f2e8baaff0beb293ab19f6615699a521b60"}
Jul 06 19:32:47.964949 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:47.964928    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rwdf" event={"ID":"398cd612-d4e2-4ac0-b8c4-c632dd73957f","Type":"ContainerStarted","Data":"436448dca563ba4fa52a9888d04707cf1a0ab87718bf179785852031cd5dafbc"}
Jul 06 19:32:47.966411 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:47.966390    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2cwdm" event={"ID":"75a74486-1003-45bb-9c55-1cbf52ab4090","Type":"ContainerStarted","Data":"2bc326c0ccda454e58767305699b122c0ea03fa471f857316707bd0e4ebcfc1c"}
Jul 06 19:32:48.627504 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:48.627456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs\") pod \"network-metrics-daemon-kbl7t\" (UID: \"2bd4ee01-7358-4e6f-9b67-43d41aef4fc3\") " pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:48.627664 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:48.627550    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret\") pod \"global-pull-secret-syncer-zgsll\" (UID: \"05300884-0a61-4b61-9637-add8ddfff1c5\") " pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:48.627664 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:48.627593    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 19:32:48.627664 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:48.627646    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 19:32:48.627664 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:48.627659    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs podName:2bd4ee01-7358-4e6f-9b67-43d41aef4fc3 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:04.627640283 +0000 UTC m=+33.168816954 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs") pod "network-metrics-daemon-kbl7t" (UID: "2bd4ee01-7358-4e6f-9b67-43d41aef4fc3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 19:32:48.627828 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:48.627679    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret podName:05300884-0a61-4b61-9637-add8ddfff1c5 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:04.627669331 +0000 UTC m=+33.168845990 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret") pod "global-pull-secret-syncer-zgsll" (UID: "05300884-0a61-4b61-9637-add8ddfff1c5") : object "kube-system"/"original-pull-secret" not registered
Jul 06 19:32:48.728029 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:48.727998    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q87t4\" (UniqueName: \"kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4\") pod \"network-check-target-lvxg5\" (UID: \"2fc30bdf-7515-45f6-9c88-2f976891430f\") " pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:48.728179 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:48.728142    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 19:32:48.728179 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:48.728160    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 19:32:48.728179 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:48.728171    2562 projected.go:196] Error preparing data for projected volume kube-api-access-q87t4 for pod openshift-network-diagnostics/network-check-target-lvxg5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 19:32:48.728319 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:48.728228    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4 podName:2fc30bdf-7515-45f6-9c88-2f976891430f nodeName:}" failed. No retries permitted until 2026-07-06 19:33:04.728210104 +0000 UTC m=+33.269386761 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q87t4" (UniqueName: "kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4") pod "network-check-target-lvxg5" (UID: "2fc30bdf-7515-45f6-9c88-2f976891430f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 19:32:48.889317 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:48.889233    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:48.889317 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:48.889250    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:48.889317 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:48.889273    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:48.889619 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:48.889365    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvxg5" podUID="2fc30bdf-7515-45f6-9c88-2f976891430f"
Jul 06 19:32:48.889619 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:48.889444    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgsll" podUID="05300884-0a61-4b61-9637-add8ddfff1c5"
Jul 06 19:32:48.889619 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:48.889564    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbl7t" podUID="2bd4ee01-7358-4e6f-9b67-43d41aef4fc3"
Jul 06 19:32:50.889318 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:50.889290    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:50.890077 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:50.889339    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:50.890077 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:50.889346    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:50.890077 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:50.889428    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgsll" podUID="05300884-0a61-4b61-9637-add8ddfff1c5"
Jul 06 19:32:50.890077 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:50.889555    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvxg5" podUID="2fc30bdf-7515-45f6-9c88-2f976891430f"
Jul 06 19:32:50.890077 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:50.889639    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbl7t" podUID="2bd4ee01-7358-4e6f-9b67-43d41aef4fc3"
Jul 06 19:32:51.134653 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:51.134590    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dfspr" podStartSLOduration=6.6072962539999995 podStartE2EDuration="19.134569349s" totalImagesPullingTime="12.527273095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:46.973259953 +0000 UTC m=+15.514436630" watchObservedRunningTime="2026-07-06 19:32:51.134569349 +0000 UTC m=+19.675746026"
Jul 06 19:32:51.134849 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:51.134737    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-n9f6w" podStartSLOduration=5.640060174 podStartE2EDuration="18.134731862s" totalImagesPullingTime="12.494671688s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:51.134044304 +0000 UTC m=+19.675220979" watchObservedRunningTime="2026-07-06 19:32:51.134731862 +0000 UTC m=+19.675908542"
Jul 06 19:32:51.155949 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:51.155060    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cklnx" podStartSLOduration=6.644201307 podStartE2EDuration="19.155034713s" totalImagesPullingTime="12.510833406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:51.154199644 +0000 UTC m=+19.695376321" watchObservedRunningTime="2026-07-06 19:32:51.155034713 +0000 UTC m=+19.696211391"
Jul 06 19:32:51.179055 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:51.179016    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-z5bg8" podStartSLOduration=5.677948983 podStartE2EDuration="18.178998989s" totalImagesPullingTime="12.501050006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:51.178524657 +0000 UTC m=+19.719701334" watchObservedRunningTime="2026-07-06 19:32:51.178998989 +0000 UTC m=+19.720175666"
Jul 06 19:32:51.379823 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:51.379758    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2cwdm" podStartSLOduration=5.905440353 podStartE2EDuration="18.379738766s" totalImagesPullingTime="12.474298413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:51.378989241 +0000 UTC m=+19.920165917" watchObservedRunningTime="2026-07-06 19:32:51.379738766 +0000 UTC m=+19.920915635"
Jul 06 19:32:51.485131 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:51.485058    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cklnx"
Jul 06 19:32:51.485887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:51.485866    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cklnx"
Jul 06 19:32:51.974104 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:51.974072    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cklnx"
Jul 06 19:32:51.974590 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:51.974568    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cklnx"
Jul 06 19:32:52.889359 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:52.889323    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:52.889359 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:52.889347    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:52.889635 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:52.889323    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:52.889635 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:52.889432    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgsll" podUID="05300884-0a61-4b61-9637-add8ddfff1c5"
Jul 06 19:32:52.889635 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:52.889511    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvxg5" podUID="2fc30bdf-7515-45f6-9c88-2f976891430f"
Jul 06 19:32:52.889635 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:52.889591    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbl7t" podUID="2bd4ee01-7358-4e6f-9b67-43d41aef4fc3"
Jul 06 19:32:54.889767 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:54.889361    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:54.890190 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:54.889534    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:54.890190 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:54.889877    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgsll" podUID="05300884-0a61-4b61-9637-add8ddfff1c5"
Jul 06 19:32:54.890190 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:54.889544    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:54.890190 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:54.889960    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbl7t" podUID="2bd4ee01-7358-4e6f-9b67-43d41aef4fc3"
Jul 06 19:32:54.890190 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:54.890013    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvxg5" podUID="2fc30bdf-7515-45f6-9c88-2f976891430f"
Jul 06 19:32:54.981260 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:54.981235    2562 generic.go:356] "Generic (PLEG): container finished" podID="cd31cf43040b67356073a3b42229e6bd" containerID="de81d4078ca2bd8248b85064d236b59de2f6bd16438add05b4f93ba62e459bb1" exitCode=0
Jul 06 19:32:54.981372 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:54.981303    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal" event={"ID":"cd31cf43040b67356073a3b42229e6bd","Type":"ContainerDied","Data":"de81d4078ca2bd8248b85064d236b59de2f6bd16438add05b4f93ba62e459bb1"}
Jul 06 19:32:54.982947 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:54.982925    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tkgjn" event={"ID":"1a262cc1-2cd2-40f0-ba32-fc612faf3c4a","Type":"ContainerStarted","Data":"a909b5a86399f624c9956b257c4651d2426a0372c06e678cab556906e061ef69"}
Jul 06 19:32:54.985944 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:54.985922    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wt558" event={"ID":"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4","Type":"ContainerStarted","Data":"9434ab1fd5af6a8dae4df1cdc78311130c544e9035955be0647a6b7fd90fbbcd"}
Jul 06 19:32:54.986031 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:54.985999    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wt558" event={"ID":"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4","Type":"ContainerStarted","Data":"bd39bb17e4b75c7838c0681f96d0be0114e4a6f9b70e439bb470b904f2dd1414"}
Jul 06 19:32:54.986073 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:54.986050    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wt558" event={"ID":"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4","Type":"ContainerStarted","Data":"a41f30d7c948fcb745c977a5a105c5461191462280d4ce89351e28278a2db6d8"}
Jul 06 19:32:54.986116 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:54.986100    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wt558" event={"ID":"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4","Type":"ContainerStarted","Data":"48947e3a9d7ed4056ccd2ba428969a41578f61d0818dd48ffc34beedc91a1227"}
Jul 06 19:32:54.986157 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:54.986150    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wt558" event={"ID":"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4","Type":"ContainerStarted","Data":"50ac80a9120cd555982cf418470017505542e3dfb9242764aa3ede3127e72414"}
Jul 06 19:32:54.986211 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:54.986197    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wt558" event={"ID":"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4","Type":"ContainerStarted","Data":"830ef34fa53da62d4fefeb48bb406e7257996fadad0022878117befac93ea4cf"}
Jul 06 19:32:54.987369 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:54.987346    2562 generic.go:356] "Generic (PLEG): container finished" podID="398cd612-d4e2-4ac0-b8c4-c632dd73957f" containerID="436448dca563ba4fa52a9888d04707cf1a0ab87718bf179785852031cd5dafbc" exitCode=0
Jul 06 19:32:54.987442 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:54.987381    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rwdf" event={"ID":"398cd612-d4e2-4ac0-b8c4-c632dd73957f","Type":"ContainerDied","Data":"436448dca563ba4fa52a9888d04707cf1a0ab87718bf179785852031cd5dafbc"}
Jul 06 19:32:55.080369 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:55.080331    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tkgjn" podStartSLOduration=2.122960292 podStartE2EDuration="23.080318148s" totalImagesPullingTime="20.957357856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:55.079512224 +0000 UTC m=+23.620688900" watchObservedRunningTime="2026-07-06 19:32:55.080318148 +0000 UTC m=+23.621494824"
Jul 06 19:32:55.858575 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:55.858553    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 06 19:32:55.868051 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:55.867979    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T19:32:55.85857231Z","UUID":"ff2422bc-b3e8-48ba-9c6c-4902e9bb40fe","Handler":null,"Name":"","Endpoint":""}
Jul 06 19:32:55.869191 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:55.869175    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 06 19:32:55.869264 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:55.869207    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 06 19:32:55.992125 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:55.992065    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal" event={"ID":"cd31cf43040b67356073a3b42229e6bd","Type":"ContainerStarted","Data":"457eff8dcd3a4ab6382aad5299b42cfd71d3d0cef09644f80776251a671174ed"}
Jul 06 19:32:55.994041 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:55.994012    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt" event={"ID":"b29855f2-a8a2-49f4-8d0b-7f52c478e852","Type":"ContainerStarted","Data":"1aba5997ce08c10d13169bab12d403f7c923edfa71d1b3c92b150551b3c7cf66"}
Jul 06 19:32:56.011704 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:56.011664    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-200.ec2.internal" podStartSLOduration=23.011650614 podStartE2EDuration="23.011650614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:56.011345888 +0000 UTC m=+24.552522564" watchObservedRunningTime="2026-07-06 19:32:56.011650614 +0000 UTC m=+24.552827289"
Jul 06 19:32:56.889084 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:56.889046    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:56.889354 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:56.889151    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:56.889354 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:56.889175    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbl7t" podUID="2bd4ee01-7358-4e6f-9b67-43d41aef4fc3"
Jul 06 19:32:56.889354 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:56.889221    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgsll" podUID="05300884-0a61-4b61-9637-add8ddfff1c5"
Jul 06 19:32:56.889354 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:56.889266    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:56.889354 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:56.889342    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvxg5" podUID="2fc30bdf-7515-45f6-9c88-2f976891430f"
Jul 06 19:32:58.001024 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:58.000991    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wt558" event={"ID":"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4","Type":"ContainerStarted","Data":"7e32935494582f37d9d7983af927b6a629d3173a9f486fe34300b526d639a4d7"}
Jul 06 19:32:58.003217 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:58.003190    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt" event={"ID":"b29855f2-a8a2-49f4-8d0b-7f52c478e852","Type":"ContainerStarted","Data":"ba7eb07e070467f086050bc516106133520b98552d044939e8170ba47decc112"}
Jul 06 19:32:58.889191 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:58.889157    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:32:58.889191 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:58.889186    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:32:58.889402 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:32:58.889187    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:32:58.889402 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:58.889280    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbl7t" podUID="2bd4ee01-7358-4e6f-9b67-43d41aef4fc3"
Jul 06 19:32:58.889402 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:58.889332    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgsll" podUID="05300884-0a61-4b61-9637-add8ddfff1c5"
Jul 06 19:32:58.889402 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:32:58.889393    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvxg5" podUID="2fc30bdf-7515-45f6-9c88-2f976891430f"
Jul 06 19:33:00.011898 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:00.011552    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wt558" event={"ID":"d2b1ab09-9428-4ccf-bcf1-fe21564d41f4","Type":"ContainerStarted","Data":"2fabcf113e2b465afe9dec0e65cfb530fa8b40896fdb3f64e43c08921700afc0"}
Jul 06 19:33:00.011898 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:00.011895    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:33:00.012416 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:00.011964    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:33:00.012416 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:00.012013    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:33:00.029160 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:00.029138    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:33:00.029428 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:00.029412    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:33:00.041640 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:00.041602    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l7fpt" podStartSLOduration=5.737137597 podStartE2EDuration="28.041587647s" totalImagesPullingTime="22.30445005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:58.029503812 +0000 UTC m=+26.570680489" watchObservedRunningTime="2026-07-06 19:33:00.041587647 +0000 UTC m=+28.582764374"
Jul 06 19:33:00.042140 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:00.042111    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wt558" podStartSLOduration=7.038800931 podStartE2EDuration="28.042098375s" totalImagesPullingTime="21.003297444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:00.041349115 +0000 UTC m=+28.582525791" watchObservedRunningTime="2026-07-06 19:33:00.042098375 +0000 UTC m=+28.583275051"
Jul 06 19:33:00.889182 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:00.889149    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:33:00.889384 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:00.889188    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:33:00.889384 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:00.889274    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:33:00.889384 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:00.889293    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbl7t" podUID="2bd4ee01-7358-4e6f-9b67-43d41aef4fc3"
Jul 06 19:33:00.889384 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:00.889355    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvxg5" podUID="2fc30bdf-7515-45f6-9c88-2f976891430f"
Jul 06 19:33:00.889582 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:00.889428    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgsll" podUID="05300884-0a61-4b61-9637-add8ddfff1c5"
Jul 06 19:33:01.496509 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:01.496455    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zgsll"]
Jul 06 19:33:01.496946 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:01.496648    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:33:01.496946 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:01.496763    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgsll" podUID="05300884-0a61-4b61-9637-add8ddfff1c5"
Jul 06 19:33:01.497346 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:01.497325    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kbl7t"]
Jul 06 19:33:01.497430 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:01.497417    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:33:01.497630 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:01.497581    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbl7t" podUID="2bd4ee01-7358-4e6f-9b67-43d41aef4fc3"
Jul 06 19:33:01.499014 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:01.498989    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lvxg5"]
Jul 06 19:33:01.499116 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:01.499072    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:33:01.499165 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:01.499152    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvxg5" podUID="2fc30bdf-7515-45f6-9c88-2f976891430f"
Jul 06 19:33:02.018478 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:02.018417    2562 generic.go:356] "Generic (PLEG): container finished" podID="398cd612-d4e2-4ac0-b8c4-c632dd73957f" containerID="bf2d12647507726819adc32c99e6f41c149bf72ea11c701417b6b13c14442b0b" exitCode=0
Jul 06 19:33:02.018595 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:02.018500    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rwdf" event={"ID":"398cd612-d4e2-4ac0-b8c4-c632dd73957f","Type":"ContainerDied","Data":"bf2d12647507726819adc32c99e6f41c149bf72ea11c701417b6b13c14442b0b"}
Jul 06 19:33:02.888662 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:02.888629    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:33:02.888662 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:02.888653    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:33:02.889153 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:02.888740    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:33:02.889153 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:02.888750    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgsll" podUID="05300884-0a61-4b61-9637-add8ddfff1c5"
Jul 06 19:33:02.889153 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:02.888860    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbl7t" podUID="2bd4ee01-7358-4e6f-9b67-43d41aef4fc3"
Jul 06 19:33:02.889153 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:02.888960    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvxg5" podUID="2fc30bdf-7515-45f6-9c88-2f976891430f"
Jul 06 19:33:04.024597 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:04.024433    2562 generic.go:356] "Generic (PLEG): container finished" podID="398cd612-d4e2-4ac0-b8c4-c632dd73957f" containerID="46a0cfc2c8cd4cfb228ce842e69bbf6170199e1abd2a672209f617e71e45655b" exitCode=0
Jul 06 19:33:04.024974 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:04.024510    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rwdf" event={"ID":"398cd612-d4e2-4ac0-b8c4-c632dd73957f","Type":"ContainerDied","Data":"46a0cfc2c8cd4cfb228ce842e69bbf6170199e1abd2a672209f617e71e45655b"}
Jul 06 19:33:04.652363 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:04.652320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs\") pod \"network-metrics-daemon-kbl7t\" (UID: \"2bd4ee01-7358-4e6f-9b67-43d41aef4fc3\") " pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:33:04.652549 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:04.652380    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret\") pod \"global-pull-secret-syncer-zgsll\" (UID: \"05300884-0a61-4b61-9637-add8ddfff1c5\") " pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:33:04.652549 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:04.652470    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 19:33:04.652549 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:04.652480    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 19:33:04.652549 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:04.652539    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret podName:05300884-0a61-4b61-9637-add8ddfff1c5 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:36.65252534 +0000 UTC m=+65.193701994 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret") pod "global-pull-secret-syncer-zgsll" (UID: "05300884-0a61-4b61-9637-add8ddfff1c5") : object "kube-system"/"original-pull-secret" not registered
Jul 06 19:33:04.652702 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:04.652556    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs podName:2bd4ee01-7358-4e6f-9b67-43d41aef4fc3 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:36.652548515 +0000 UTC m=+65.193725169 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs") pod "network-metrics-daemon-kbl7t" (UID: "2bd4ee01-7358-4e6f-9b67-43d41aef4fc3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 19:33:04.753013 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:04.752984    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q87t4\" (UniqueName: \"kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4\") pod \"network-check-target-lvxg5\" (UID: \"2fc30bdf-7515-45f6-9c88-2f976891430f\") " pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:33:04.753137 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:04.753101    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 19:33:04.753137 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:04.753117    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 19:33:04.753137 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:04.753127    2562 projected.go:196] Error preparing data for projected volume kube-api-access-q87t4 for pod openshift-network-diagnostics/network-check-target-lvxg5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 19:33:04.753266 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:04.753170    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4 podName:2fc30bdf-7515-45f6-9c88-2f976891430f nodeName:}" failed. No retries permitted until 2026-07-06 19:33:36.75316 +0000 UTC m=+65.294336654 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-q87t4" (UniqueName: "kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4") pod "network-check-target-lvxg5" (UID: "2fc30bdf-7515-45f6-9c88-2f976891430f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 19:33:04.888726 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:04.888703    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:33:04.888836 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:04.888793    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:33:04.888836 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:04.888800    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:33:04.888836 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:04.888797    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgsll" podUID="05300884-0a61-4b61-9637-add8ddfff1c5"
Jul 06 19:33:04.888939 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:04.888881    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbl7t" podUID="2bd4ee01-7358-4e6f-9b67-43d41aef4fc3"
Jul 06 19:33:04.888971 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:04.888948    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvxg5" podUID="2fc30bdf-7515-45f6-9c88-2f976891430f"
Jul 06 19:33:06.030741 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:06.030714    2562 generic.go:356] "Generic (PLEG): container finished" podID="398cd612-d4e2-4ac0-b8c4-c632dd73957f" containerID="8ca471bc71cc12e097917b2c876de81f415e89f89f11ea54bac5c7038580b132" exitCode=0
Jul 06 19:33:06.031335 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:06.030771    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rwdf" event={"ID":"398cd612-d4e2-4ac0-b8c4-c632dd73957f","Type":"ContainerDied","Data":"8ca471bc71cc12e097917b2c876de81f415e89f89f11ea54bac5c7038580b132"}
Jul 06 19:33:06.889366 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:06.889330    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:33:06.889549 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:06.889427    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:33:06.889549 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:06.889435    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvxg5" podUID="2fc30bdf-7515-45f6-9c88-2f976891430f"
Jul 06 19:33:06.889549 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:06.889535    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kbl7t" podUID="2bd4ee01-7358-4e6f-9b67-43d41aef4fc3"
Jul 06 19:33:06.889669 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:06.889592    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:33:06.889669 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:06.889649    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgsll" podUID="05300884-0a61-4b61-9637-add8ddfff1c5"
Jul 06 19:33:07.309794 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.309770    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-200.ec2.internal" event="NodeReady"
Jul 06 19:33:07.310221 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.309896    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 06 19:33:07.349173 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.349127    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7788f56567-99q8x"]
Jul 06 19:33:07.376432 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.376409    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-phrpj"]
Jul 06 19:33:07.376586 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.376562    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.379860 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.379838    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 06 19:33:07.379860 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.379849    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 06 19:33:07.380008 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.379838    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rqjk7\""
Jul 06 19:33:07.380144 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.380131    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 06 19:33:07.385696 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.385676    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 06 19:33:07.394412 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.394392    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n6x2q"]
Jul 06 19:33:07.394690 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.394673    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-phrpj"
Jul 06 19:33:07.399688 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.399671    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 06 19:33:07.399688 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.399682    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 06 19:33:07.399941 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.399925    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zq5lq\""
Jul 06 19:33:07.408699 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.408675    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7788f56567-99q8x"]
Jul 06 19:33:07.408798 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.408706    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n6x2q"]
Jul 06 19:33:07.408798 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.408720    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-phrpj"]
Jul 06 19:33:07.408798 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.408783    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n6x2q"
Jul 06 19:33:07.412155 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.412133    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 06 19:33:07.412269 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.412250    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c64kp\""
Jul 06 19:33:07.412476 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.412454    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 06 19:33:07.412575 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.412566    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 06 19:33:07.474984 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.474961    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.475092 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.475001    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j95kw\" (UniqueName: \"kubernetes.io/projected/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-kube-api-access-j95kw\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:33:07.475092 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.475028    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-certificates\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.475092 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.475066    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:33:07.475256 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.475118    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99cfd3ef-94a2-464d-aca2-561f8e30939e-trusted-ca\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.475256 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.475163    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-tmp-dir\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:33:07.475256 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.475191    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnfbc\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-kube-api-access-bnfbc\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.475256 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.475221    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99cfd3ef-94a2-464d-aca2-561f8e30939e-image-registry-private-configuration\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.475256 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.475246    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99cfd3ef-94a2-464d-aca2-561f8e30939e-installation-pull-secrets\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.475442 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.475269    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-bound-sa-token\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.475442 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.475308    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99cfd3ef-94a2-464d-aca2-561f8e30939e-ca-trust-extracted\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.475442 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.475345    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-config-volume\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:33:07.555750 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.555718    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:07.575666 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.575597    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j95kw\" (UniqueName: \"kubernetes.io/projected/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-kube-api-access-j95kw\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:33:07.575666 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.575639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-certificates\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.575798 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.575669    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:33:07.575798 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.575691    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99cfd3ef-94a2-464d-aca2-561f8e30939e-trusted-ca\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.575798 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.575719    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-tmp-dir\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:33:07.575798 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.575742    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnfbc\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-kube-api-access-bnfbc\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.575984 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:07.575882    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 19:33:07.576066 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:07.576050    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls podName:44bd8f04-80fc-4f99-97f3-3f84666ae1d0 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:08.076009099 +0000 UTC m=+36.617185768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls") pod "dns-default-phrpj" (UID: "44bd8f04-80fc-4f99-97f3-3f84666ae1d0") : secret "dns-default-metrics-tls" not found
Jul 06 19:33:07.576130 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.576082    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99cfd3ef-94a2-464d-aca2-561f8e30939e-image-registry-private-configuration\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.576188 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.576150    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99cfd3ef-94a2-464d-aca2-561f8e30939e-installation-pull-secrets\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.576188 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.576178    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-bound-sa-token\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.576282 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.576260    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99cfd3ef-94a2-464d-aca2-561f8e30939e-ca-trust-extracted\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.576282 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.576272    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-tmp-dir\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:33:07.576370 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.576309    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-certificates\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.576370 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.576345    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-config-volume\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:33:07.576461 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.576409    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d8e171c-5f31-43a4-ad00-034d98242a89-cert\") pod \"ingress-canary-n6x2q\" (UID: \"3d8e171c-5f31-43a4-ad00-034d98242a89\") " pod="openshift-ingress-canary/ingress-canary-n6x2q"
Jul 06 19:33:07.576544 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.576527    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whz95\" (UniqueName: \"kubernetes.io/projected/3d8e171c-5f31-43a4-ad00-034d98242a89-kube-api-access-whz95\") pod \"ingress-canary-n6x2q\" (UID: \"3d8e171c-5f31-43a4-ad00-034d98242a89\") " pod="openshift-ingress-canary/ingress-canary-n6x2q"
Jul 06 19:33:07.576600 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.576575    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.576775 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:07.576760    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 19:33:07.576822 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:07.576779    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7788f56567-99q8x: secret "image-registry-tls" not found
Jul 06 19:33:07.576856 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:07.576825    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls podName:99cfd3ef-94a2-464d-aca2-561f8e30939e nodeName:}" failed. No retries permitted until 2026-07-06 19:33:08.076809191 +0000 UTC m=+36.617985856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls") pod "image-registry-7788f56567-99q8x" (UID: "99cfd3ef-94a2-464d-aca2-561f8e30939e") : secret "image-registry-tls" not found
Jul 06 19:33:07.576856 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.576838    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99cfd3ef-94a2-464d-aca2-561f8e30939e-trusted-ca\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.576999 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.576976    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-config-volume\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:33:07.577193 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.577152    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99cfd3ef-94a2-464d-aca2-561f8e30939e-ca-trust-extracted\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.581648 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.581627    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99cfd3ef-94a2-464d-aca2-561f8e30939e-image-registry-private-configuration\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.581716 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.581653    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99cfd3ef-94a2-464d-aca2-561f8e30939e-installation-pull-secrets\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.585562 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.585541    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-bound-sa-token\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.585902 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.585883    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j95kw\" (UniqueName: \"kubernetes.io/projected/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-kube-api-access-j95kw\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:33:07.586013 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.585997    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnfbc\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-kube-api-access-bnfbc\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:07.671468 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.671439    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:07.677409 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.677391    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d8e171c-5f31-43a4-ad00-034d98242a89-cert\") pod \"ingress-canary-n6x2q\" (UID: \"3d8e171c-5f31-43a4-ad00-034d98242a89\") " pod="openshift-ingress-canary/ingress-canary-n6x2q"
Jul 06 19:33:07.677477 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.677429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whz95\" (UniqueName: \"kubernetes.io/projected/3d8e171c-5f31-43a4-ad00-034d98242a89-kube-api-access-whz95\") pod \"ingress-canary-n6x2q\" (UID: \"3d8e171c-5f31-43a4-ad00-034d98242a89\") " pod="openshift-ingress-canary/ingress-canary-n6x2q"
Jul 06 19:33:07.679337 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.679319    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d8e171c-5f31-43a4-ad00-034d98242a89-cert\") pod \"ingress-canary-n6x2q\" (UID: \"3d8e171c-5f31-43a4-ad00-034d98242a89\") " pod="openshift-ingress-canary/ingress-canary-n6x2q"
Jul 06 19:33:07.687465 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.687440    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whz95\" (UniqueName: \"kubernetes.io/projected/3d8e171c-5f31-43a4-ad00-034d98242a89-kube-api-access-whz95\") pod \"ingress-canary-n6x2q\" (UID: \"3d8e171c-5f31-43a4-ad00-034d98242a89\") " pod="openshift-ingress-canary/ingress-canary-n6x2q"
Jul 06 19:33:07.718544 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.718525    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n6x2q"
Jul 06 19:33:07.914214 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:07.914045    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n6x2q"]
Jul 06 19:33:07.918139 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:33:07.918115    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8e171c_5f31_43a4_ad00_034d98242a89.slice/crio-dcab2291397acacc2c34affd7714d5fe250598aa5fde0026dd4e8d71dd7813ba WatchSource:0}: Error finding container dcab2291397acacc2c34affd7714d5fe250598aa5fde0026dd4e8d71dd7813ba: Status 404 returned error can't find the container with id dcab2291397acacc2c34affd7714d5fe250598aa5fde0026dd4e8d71dd7813ba
Jul 06 19:33:08.037405 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:08.037370    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n6x2q" event={"ID":"3d8e171c-5f31-43a4-ad00-034d98242a89","Type":"ContainerStarted","Data":"dcab2291397acacc2c34affd7714d5fe250598aa5fde0026dd4e8d71dd7813ba"}
Jul 06 19:33:08.080742 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:08.080716    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:08.080840 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:08.080758    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:33:08.080899 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:08.080853    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 19:33:08.080899 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:08.080868    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7788f56567-99q8x: secret "image-registry-tls" not found
Jul 06 19:33:08.080997 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:08.080906    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 19:33:08.080997 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:08.080931    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls podName:99cfd3ef-94a2-464d-aca2-561f8e30939e nodeName:}" failed. No retries permitted until 2026-07-06 19:33:09.080915422 +0000 UTC m=+37.622092076 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls") pod "image-registry-7788f56567-99q8x" (UID: "99cfd3ef-94a2-464d-aca2-561f8e30939e") : secret "image-registry-tls" not found
Jul 06 19:33:08.080997 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:08.080967    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls podName:44bd8f04-80fc-4f99-97f3-3f84666ae1d0 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:09.080950464 +0000 UTC m=+37.622127125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls") pod "dns-default-phrpj" (UID: "44bd8f04-80fc-4f99-97f3-3f84666ae1d0") : secret "dns-default-metrics-tls" not found
Jul 06 19:33:08.888782 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:08.888747    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:33:08.889464 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:08.888747    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:33:08.889464 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:08.888747    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:33:08.892083 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:08.892065    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 19:33:08.893445 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:08.893426    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b9cds\""
Jul 06 19:33:08.893564 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:08.893473    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 19:33:08.893625 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:08.893426    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4gp52\""
Jul 06 19:33:08.893717 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:08.893698    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 19:33:08.893776 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:08.893750    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 19:33:09.089761 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.089735    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:09.089761 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.089766    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:33:09.089971 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:09.089900    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 19:33:09.089971 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:09.089925    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7788f56567-99q8x: secret "image-registry-tls" not found
Jul 06 19:33:09.089971 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:09.089907    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 19:33:09.090120 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:09.089991    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls podName:99cfd3ef-94a2-464d-aca2-561f8e30939e nodeName:}" failed. No retries permitted until 2026-07-06 19:33:11.089971625 +0000 UTC m=+39.631148280 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls") pod "image-registry-7788f56567-99q8x" (UID: "99cfd3ef-94a2-464d-aca2-561f8e30939e") : secret "image-registry-tls" not found
Jul 06 19:33:09.090120 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:09.090012    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls podName:44bd8f04-80fc-4f99-97f3-3f84666ae1d0 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:11.090002673 +0000 UTC m=+39.631179348 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls") pod "dns-default-phrpj" (UID: "44bd8f04-80fc-4f99-97f3-3f84666ae1d0") : secret "dns-default-metrics-tls" not found
Jul 06 19:33:09.264779 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.264754    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69978bdcb4-7jh7j"]
Jul 06 19:33:09.286297 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.286274    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684c78ffcd-xz4z8"]
Jul 06 19:33:09.286419 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.286398    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69978bdcb4-7jh7j"
Jul 06 19:33:09.289554 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.289417    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 06 19:33:09.289554 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.289441    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 06 19:33:09.289722 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.289555    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 06 19:33:09.289722 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.289632    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 06 19:33:09.310414 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.310390    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"]
Jul 06 19:33:09.310547 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.310390    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684c78ffcd-xz4z8"
Jul 06 19:33:09.314996 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.314803    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-s46bn\""
Jul 06 19:33:09.314996 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.314831    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 06 19:33:09.326419 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.326344    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69978bdcb4-7jh7j"]
Jul 06 19:33:09.326539 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.326437    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.326539 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.326473    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"]
Jul 06 19:33:09.326539 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.326522    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684c78ffcd-xz4z8"]
Jul 06 19:33:09.329878 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.329855    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 06 19:33:09.330997 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.330814    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 06 19:33:09.330997 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.330909    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 06 19:33:09.331156 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.331014    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 06 19:33:09.391326 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.391306    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdwh8\" (UniqueName: \"kubernetes.io/projected/7dc3dea6-dcba-43a1-9708-9fce1f1e3f90-kube-api-access-sdwh8\") pod \"managed-serviceaccount-addon-agent-684c78ffcd-xz4z8\" (UID: \"7dc3dea6-dcba-43a1-9708-9fce1f1e3f90\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684c78ffcd-xz4z8"
Jul 06 19:33:09.391447 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.391333    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f0b8344-b8bc-4820-905f-3d7a6da97e0a-tmp\") pod \"klusterlet-addon-workmgr-69978bdcb4-7jh7j\" (UID: \"5f0b8344-b8bc-4820-905f-3d7a6da97e0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69978bdcb4-7jh7j"
Jul 06 19:33:09.391447 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.391358    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7dc3dea6-dcba-43a1-9708-9fce1f1e3f90-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-684c78ffcd-xz4z8\" (UID: \"7dc3dea6-dcba-43a1-9708-9fce1f1e3f90\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684c78ffcd-xz4z8"
Jul 06 19:33:09.391447 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.391428    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5f0b8344-b8bc-4820-905f-3d7a6da97e0a-klusterlet-config\") pod \"klusterlet-addon-workmgr-69978bdcb4-7jh7j\" (UID: \"5f0b8344-b8bc-4820-905f-3d7a6da97e0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69978bdcb4-7jh7j"
Jul 06 19:33:09.391637 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.391464    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7vxz\" (UniqueName: \"kubernetes.io/projected/5f0b8344-b8bc-4820-905f-3d7a6da97e0a-kube-api-access-k7vxz\") pod \"klusterlet-addon-workmgr-69978bdcb4-7jh7j\" (UID: \"5f0b8344-b8bc-4820-905f-3d7a6da97e0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69978bdcb4-7jh7j"
Jul 06 19:33:09.492637 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.492616    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e79108f0-9a0a-49a4-a720-86af45b7d6bb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59c4585f7f-9lnns\" (UID: \"e79108f0-9a0a-49a4-a720-86af45b7d6bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.492751 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.492669    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x6gv\" (UniqueName: \"kubernetes.io/projected/e79108f0-9a0a-49a4-a720-86af45b7d6bb-kube-api-access-4x6gv\") pod \"cluster-proxy-proxy-agent-59c4585f7f-9lnns\" (UID: \"e79108f0-9a0a-49a4-a720-86af45b7d6bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.492751 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.492707    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdwh8\" (UniqueName: \"kubernetes.io/projected/7dc3dea6-dcba-43a1-9708-9fce1f1e3f90-kube-api-access-sdwh8\") pod \"managed-serviceaccount-addon-agent-684c78ffcd-xz4z8\" (UID: \"7dc3dea6-dcba-43a1-9708-9fce1f1e3f90\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684c78ffcd-xz4z8"
Jul 06 19:33:09.492872 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.492745    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f0b8344-b8bc-4820-905f-3d7a6da97e0a-tmp\") pod \"klusterlet-addon-workmgr-69978bdcb4-7jh7j\" (UID: \"5f0b8344-b8bc-4820-905f-3d7a6da97e0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69978bdcb4-7jh7j"
Jul 06 19:33:09.492872 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.492851    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7dc3dea6-dcba-43a1-9708-9fce1f1e3f90-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-684c78ffcd-xz4z8\" (UID: \"7dc3dea6-dcba-43a1-9708-9fce1f1e3f90\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684c78ffcd-xz4z8"
Jul 06 19:33:09.492965 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.492899    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e79108f0-9a0a-49a4-a720-86af45b7d6bb-ca\") pod \"cluster-proxy-proxy-agent-59c4585f7f-9lnns\" (UID: \"e79108f0-9a0a-49a4-a720-86af45b7d6bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.492965 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.492928    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e79108f0-9a0a-49a4-a720-86af45b7d6bb-hub\") pod \"cluster-proxy-proxy-agent-59c4585f7f-9lnns\" (UID: \"e79108f0-9a0a-49a4-a720-86af45b7d6bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.493061 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.492984    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5f0b8344-b8bc-4820-905f-3d7a6da97e0a-klusterlet-config\") pod \"klusterlet-addon-workmgr-69978bdcb4-7jh7j\" (UID: \"5f0b8344-b8bc-4820-905f-3d7a6da97e0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69978bdcb4-7jh7j"
Jul 06 19:33:09.493061 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.493029    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e79108f0-9a0a-49a4-a720-86af45b7d6bb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59c4585f7f-9lnns\" (UID: \"e79108f0-9a0a-49a4-a720-86af45b7d6bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.493146 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.493077    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7vxz\" (UniqueName: \"kubernetes.io/projected/5f0b8344-b8bc-4820-905f-3d7a6da97e0a-kube-api-access-k7vxz\") pod \"klusterlet-addon-workmgr-69978bdcb4-7jh7j\" (UID: \"5f0b8344-b8bc-4820-905f-3d7a6da97e0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69978bdcb4-7jh7j"
Jul 06 19:33:09.493146 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.493117    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e79108f0-9a0a-49a4-a720-86af45b7d6bb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59c4585f7f-9lnns\" (UID: \"e79108f0-9a0a-49a4-a720-86af45b7d6bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.493146 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.493124    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f0b8344-b8bc-4820-905f-3d7a6da97e0a-tmp\") pod \"klusterlet-addon-workmgr-69978bdcb4-7jh7j\" (UID: \"5f0b8344-b8bc-4820-905f-3d7a6da97e0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69978bdcb4-7jh7j"
Jul 06 19:33:09.496591 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.496563    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5f0b8344-b8bc-4820-905f-3d7a6da97e0a-klusterlet-config\") pod \"klusterlet-addon-workmgr-69978bdcb4-7jh7j\" (UID: \"5f0b8344-b8bc-4820-905f-3d7a6da97e0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69978bdcb4-7jh7j"
Jul 06 19:33:09.496674 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.496569    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7dc3dea6-dcba-43a1-9708-9fce1f1e3f90-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-684c78ffcd-xz4z8\" (UID: \"7dc3dea6-dcba-43a1-9708-9fce1f1e3f90\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684c78ffcd-xz4z8"
Jul 06 19:33:09.503424 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.503398    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdwh8\" (UniqueName: \"kubernetes.io/projected/7dc3dea6-dcba-43a1-9708-9fce1f1e3f90-kube-api-access-sdwh8\") pod \"managed-serviceaccount-addon-agent-684c78ffcd-xz4z8\" (UID: \"7dc3dea6-dcba-43a1-9708-9fce1f1e3f90\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684c78ffcd-xz4z8"
Jul 06 19:33:09.503608 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.503398    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7vxz\" (UniqueName: \"kubernetes.io/projected/5f0b8344-b8bc-4820-905f-3d7a6da97e0a-kube-api-access-k7vxz\") pod \"klusterlet-addon-workmgr-69978bdcb4-7jh7j\" (UID: \"5f0b8344-b8bc-4820-905f-3d7a6da97e0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69978bdcb4-7jh7j"
Jul 06 19:33:09.593738 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.593470    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e79108f0-9a0a-49a4-a720-86af45b7d6bb-ca\") pod \"cluster-proxy-proxy-agent-59c4585f7f-9lnns\" (UID: \"e79108f0-9a0a-49a4-a720-86af45b7d6bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.593738 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.593525    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e79108f0-9a0a-49a4-a720-86af45b7d6bb-hub\") pod \"cluster-proxy-proxy-agent-59c4585f7f-9lnns\" (UID: \"e79108f0-9a0a-49a4-a720-86af45b7d6bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.593738 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.593558    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e79108f0-9a0a-49a4-a720-86af45b7d6bb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59c4585f7f-9lnns\" (UID: \"e79108f0-9a0a-49a4-a720-86af45b7d6bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.593738 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.593616    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e79108f0-9a0a-49a4-a720-86af45b7d6bb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59c4585f7f-9lnns\" (UID: \"e79108f0-9a0a-49a4-a720-86af45b7d6bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.593738 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.593649    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e79108f0-9a0a-49a4-a720-86af45b7d6bb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59c4585f7f-9lnns\" (UID: \"e79108f0-9a0a-49a4-a720-86af45b7d6bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.593738 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.593690    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x6gv\" (UniqueName: \"kubernetes.io/projected/e79108f0-9a0a-49a4-a720-86af45b7d6bb-kube-api-access-4x6gv\") pod \"cluster-proxy-proxy-agent-59c4585f7f-9lnns\" (UID: \"e79108f0-9a0a-49a4-a720-86af45b7d6bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.594650 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.594626    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e79108f0-9a0a-49a4-a720-86af45b7d6bb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59c4585f7f-9lnns\" (UID: \"e79108f0-9a0a-49a4-a720-86af45b7d6bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.596451 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.596425    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e79108f0-9a0a-49a4-a720-86af45b7d6bb-ca\") pod \"cluster-proxy-proxy-agent-59c4585f7f-9lnns\" (UID: \"e79108f0-9a0a-49a4-a720-86af45b7d6bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.597345 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.596904    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e79108f0-9a0a-49a4-a720-86af45b7d6bb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59c4585f7f-9lnns\" (UID: \"e79108f0-9a0a-49a4-a720-86af45b7d6bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.597345 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.597308    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e79108f0-9a0a-49a4-a720-86af45b7d6bb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59c4585f7f-9lnns\" (UID: \"e79108f0-9a0a-49a4-a720-86af45b7d6bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.597737 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.597717    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e79108f0-9a0a-49a4-a720-86af45b7d6bb-hub\") pod \"cluster-proxy-proxy-agent-59c4585f7f-9lnns\" (UID: \"e79108f0-9a0a-49a4-a720-86af45b7d6bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.598435 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.598381    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69978bdcb4-7jh7j"
Jul 06 19:33:09.608885 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.608863    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x6gv\" (UniqueName: \"kubernetes.io/projected/e79108f0-9a0a-49a4-a720-86af45b7d6bb-kube-api-access-4x6gv\") pod \"cluster-proxy-proxy-agent-59c4585f7f-9lnns\" (UID: \"e79108f0-9a0a-49a4-a720-86af45b7d6bb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.632640 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.632392    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684c78ffcd-xz4z8"
Jul 06 19:33:09.641416 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.640880    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:33:09.826070 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.825846    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69978bdcb4-7jh7j"]
Jul 06 19:33:09.829515 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.829126    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684c78ffcd-xz4z8"]
Jul 06 19:33:09.832376 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:33:09.832354    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7dc3dea6_dcba_43a1_9708_9fce1f1e3f90.slice/crio-a04df3db43ea3aa310fc2f52c05a77bc0ca3ae09747948b81ed498a301895a96 WatchSource:0}: Error finding container a04df3db43ea3aa310fc2f52c05a77bc0ca3ae09747948b81ed498a301895a96: Status 404 returned error can't find the container with id a04df3db43ea3aa310fc2f52c05a77bc0ca3ae09747948b81ed498a301895a96
Jul 06 19:33:09.859152 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:09.859129    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"]
Jul 06 19:33:09.862104 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:33:09.862067    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode79108f0_9a0a_49a4_a720_86af45b7d6bb.slice/crio-96a6a2137dbd15d838b3b3bcb52fc313b8b1204f6c12f62a01f0d8e289ef22ff WatchSource:0}: Error finding container 96a6a2137dbd15d838b3b3bcb52fc313b8b1204f6c12f62a01f0d8e289ef22ff: Status 404 returned error can't find the container with id 96a6a2137dbd15d838b3b3bcb52fc313b8b1204f6c12f62a01f0d8e289ef22ff
Jul 06 19:33:10.043844 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:10.043807    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684c78ffcd-xz4z8" event={"ID":"7dc3dea6-dcba-43a1-9708-9fce1f1e3f90","Type":"ContainerStarted","Data":"a04df3db43ea3aa310fc2f52c05a77bc0ca3ae09747948b81ed498a301895a96"}
Jul 06 19:33:10.045063 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:10.045035    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns" event={"ID":"e79108f0-9a0a-49a4-a720-86af45b7d6bb","Type":"ContainerStarted","Data":"96a6a2137dbd15d838b3b3bcb52fc313b8b1204f6c12f62a01f0d8e289ef22ff"}
Jul 06 19:33:10.046224 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:10.046200    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69978bdcb4-7jh7j" event={"ID":"5f0b8344-b8bc-4820-905f-3d7a6da97e0a","Type":"ContainerStarted","Data":"197abdbb6bb0c37aa6bc4bba75ec5a648ea7124334b0011eb85352f9a1fee7a5"}
Jul 06 19:33:11.106086 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:11.106055    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:11.106524 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:11.106094    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:33:11.106524 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:11.106232    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 19:33:11.106524 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:11.106252    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7788f56567-99q8x: secret "image-registry-tls" not found
Jul 06 19:33:11.106524 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:11.106306    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls podName:99cfd3ef-94a2-464d-aca2-561f8e30939e nodeName:}" failed. No retries permitted until 2026-07-06 19:33:15.106289242 +0000 UTC m=+43.647465899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls") pod "image-registry-7788f56567-99q8x" (UID: "99cfd3ef-94a2-464d-aca2-561f8e30939e") : secret "image-registry-tls" not found
Jul 06 19:33:11.106524 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:11.106232    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 19:33:11.106524 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:11.106387    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls podName:44bd8f04-80fc-4f99-97f3-3f84666ae1d0 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:15.106361287 +0000 UTC m=+43.647537964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls") pod "dns-default-phrpj" (UID: "44bd8f04-80fc-4f99-97f3-3f84666ae1d0") : secret "dns-default-metrics-tls" not found
Jul 06 19:33:12.672474 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:12.672438    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:12.776048 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:12.776019    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:14.114457 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:14.114261    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:14.211390 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:14.211359    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:14.608261 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:14.608142    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:14.688164 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:14.688117    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:14.822787 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:14.822751    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:15.059764 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:15.059731    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n6x2q" event={"ID":"3d8e171c-5f31-43a4-ad00-034d98242a89","Type":"ContainerStarted","Data":"dba72c50f841d2da91a82482f08016daff795a129637d1d81d447fae7e66f346"}
Jul 06 19:33:15.062320 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:15.062299    2562 generic.go:356] "Generic (PLEG): container finished" podID="398cd612-d4e2-4ac0-b8c4-c632dd73957f" containerID="0345813edef294a4c5d253a8abd519e35383d57226493a12e1ea4c34d058e8fe" exitCode=0
Jul 06 19:33:15.062427 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:15.062330    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rwdf" event={"ID":"398cd612-d4e2-4ac0-b8c4-c632dd73957f","Type":"ContainerDied","Data":"0345813edef294a4c5d253a8abd519e35383d57226493a12e1ea4c34d058e8fe"}
Jul 06 19:33:15.079630 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:15.079588    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n6x2q" podStartSLOduration=1.959327299 podStartE2EDuration="8.079574908s" totalImagesPullingTime="6.120247609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:15.078351195 +0000 UTC m=+43.619527871" watchObservedRunningTime="2026-07-06 19:33:15.079574908 +0000 UTC m=+43.620751575"
Jul 06 19:33:15.110572 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:15.110529    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:15.139622 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:15.139582    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:15.140104 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:15.139621    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:33:15.140104 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:15.139727    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 19:33:15.140104 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:15.139742    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7788f56567-99q8x: secret "image-registry-tls" not found
Jul 06 19:33:15.140104 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:15.139745    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 19:33:15.140104 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:15.139791    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls podName:99cfd3ef-94a2-464d-aca2-561f8e30939e nodeName:}" failed. No retries permitted until 2026-07-06 19:33:23.139777289 +0000 UTC m=+51.680953943 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls") pod "image-registry-7788f56567-99q8x" (UID: "99cfd3ef-94a2-464d-aca2-561f8e30939e") : secret "image-registry-tls" not found
Jul 06 19:33:15.140104 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:15.139804    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls podName:44bd8f04-80fc-4f99-97f3-3f84666ae1d0 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:23.13979789 +0000 UTC m=+51.680974543 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls") pod "dns-default-phrpj" (UID: "44bd8f04-80fc-4f99-97f3-3f84666ae1d0") : secret "dns-default-metrics-tls" not found
Jul 06 19:33:15.483591 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:15.483468    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:15.552850 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:15.552769    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:16.058132 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:16.058094    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:18.071796 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:18.071575    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69978bdcb4-7jh7j" event={"ID":"5f0b8344-b8bc-4820-905f-3d7a6da97e0a","Type":"ContainerStarted","Data":"9071b22af24c24cddc45c4e825a5549c6ca24004623755da58eee222447125c6"}
Jul 06 19:33:18.072305 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:18.071836    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69978bdcb4-7jh7j"
Jul 06 19:33:18.073129 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:18.073093    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684c78ffcd-xz4z8" event={"ID":"7dc3dea6-dcba-43a1-9708-9fce1f1e3f90","Type":"ContainerStarted","Data":"ebaeb89ff095e1f40d3c152a5626e116a4e22522b86f7c10434171a112258a4e"}
Jul 06 19:33:18.073567 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:18.073537    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69978bdcb4-7jh7j"
Jul 06 19:33:18.076171 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:18.076152    2562 generic.go:356] "Generic (PLEG): container finished" podID="398cd612-d4e2-4ac0-b8c4-c632dd73957f" containerID="7372f39df82d52c42fb264848371f83b465236860fa42285cdd0a7c21a3a8f1c" exitCode=0
Jul 06 19:33:18.076288 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:18.076223    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rwdf" event={"ID":"398cd612-d4e2-4ac0-b8c4-c632dd73957f","Type":"ContainerDied","Data":"7372f39df82d52c42fb264848371f83b465236860fa42285cdd0a7c21a3a8f1c"}
Jul 06 19:33:18.078154 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:18.078135    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns" event={"ID":"e79108f0-9a0a-49a4-a720-86af45b7d6bb","Type":"ContainerStarted","Data":"d734ea273b821f861d55a3dab5f271465a9254ad186d95981f54d568a22b78c3"}
Jul 06 19:33:18.078239 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:18.078176    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns" event={"ID":"e79108f0-9a0a-49a4-a720-86af45b7d6bb","Type":"ContainerStarted","Data":"ff0e1f617bb58e001de8a33a844c8e8b76a847cb4ae1892b4f7038fda376cbdb"}
Jul 06 19:33:18.078239 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:18.078191    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns" event={"ID":"e79108f0-9a0a-49a4-a720-86af45b7d6bb","Type":"ContainerStarted","Data":"1a5c9e087a702cc01c58dba11800e001a97296deb6f9b104b43646d37448fa11"}
Jul 06 19:33:18.092065 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:18.092030    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69978bdcb4-7jh7j" podStartSLOduration=1.276283943 podStartE2EDuration="9.092017118s" totalImagesPullingTime="7.815733175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:18.09062182 +0000 UTC m=+46.631798497" watchObservedRunningTime="2026-07-06 19:33:18.092017118 +0000 UTC m=+46.633193813"
Jul 06 19:33:18.131806 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:18.131770    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-684c78ffcd-xz4z8" podStartSLOduration=1.34223121 podStartE2EDuration="9.131758126s" totalImagesPullingTime="7.789526916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:18.131050606 +0000 UTC m=+46.672227283" watchObservedRunningTime="2026-07-06 19:33:18.131758126 +0000 UTC m=+46.672934802"
Jul 06 19:33:18.151099 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:18.151055    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns" podStartSLOduration=1.369569048 podStartE2EDuration="9.151042165s" totalImagesPullingTime="7.781473117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:18.1502856 +0000 UTC m=+46.691462276" watchObservedRunningTime="2026-07-06 19:33:18.151042165 +0000 UTC m=+46.692218840"
Jul 06 19:33:19.085113 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:19.085077    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rwdf" event={"ID":"398cd612-d4e2-4ac0-b8c4-c632dd73957f","Type":"ContainerStarted","Data":"c9a0f282f9ed4b4a883b114aa5e8adffb5577f370561da91e70ed89a41c5ad32"}
Jul 06 19:33:19.111858 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:19.111813    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5rwdf" podStartSLOduration=16.361012478 podStartE2EDuration="46.111794986s" totalImagesPullingTime="29.750782508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:19.111175562 +0000 UTC m=+47.652352238" watchObservedRunningTime="2026-07-06 19:33:19.111794986 +0000 UTC m=+47.652971661"
Jul 06 19:33:19.232339 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:19.232316    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:19.334195 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:19.334172    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:20.679695 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:20.679617    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:20.744634 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:20.744602    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:23.197737 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:23.197704    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:23.197737 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:23.197738    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:33:23.198223 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:23.197840    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 19:33:23.198223 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:23.197872    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 19:33:23.198223 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:23.197894    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls podName:44bd8f04-80fc-4f99-97f3-3f84666ae1d0 nodeName:}" failed. No retries permitted until 2026-07-06 19:33:39.197880803 +0000 UTC m=+67.739057458 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls") pod "dns-default-phrpj" (UID: "44bd8f04-80fc-4f99-97f3-3f84666ae1d0") : secret "dns-default-metrics-tls" not found
Jul 06 19:33:23.198223 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:23.197898    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7788f56567-99q8x: secret "image-registry-tls" not found
Jul 06 19:33:23.198223 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:23.197961    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls podName:99cfd3ef-94a2-464d-aca2-561f8e30939e nodeName:}" failed. No retries permitted until 2026-07-06 19:33:39.19794146 +0000 UTC m=+67.739118122 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls") pod "image-registry-7788f56567-99q8x" (UID: "99cfd3ef-94a2-464d-aca2-561f8e30939e") : secret "image-registry-tls" not found
Jul 06 19:33:25.359662 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:25.359623    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:25.459712 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:25.459681    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:27.732727 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:27.732693    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:27.834537 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:27.834508    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:30.202913 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:30.202876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:30.303373 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:30.303341    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:30.814788 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:30.814758    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:30.917874 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:30.917847    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:32.034667 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:32.034643    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wt558"
Jul 06 19:33:32.196656 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:32.196626    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:32.298175 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:32.298099    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:36.681760 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:36.681718    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs\") pod \"network-metrics-daemon-kbl7t\" (UID: \"2bd4ee01-7358-4e6f-9b67-43d41aef4fc3\") " pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:33:36.681760 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:36.681780    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret\") pod \"global-pull-secret-syncer-zgsll\" (UID: \"05300884-0a61-4b61-9637-add8ddfff1c5\") " pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:33:36.685935 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:36.685914    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 19:33:36.686010 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:36.685971    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 19:33:36.692626 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:36.692603    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 06 19:33:36.692740 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:36.692725    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs podName:2bd4ee01-7358-4e6f-9b67-43d41aef4fc3 nodeName:}" failed. No retries permitted until 2026-07-06 19:34:40.69270221 +0000 UTC m=+129.233878878 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs") pod "network-metrics-daemon-kbl7t" (UID: "2bd4ee01-7358-4e6f-9b67-43d41aef4fc3") : secret "metrics-daemon-secret" not found
Jul 06 19:33:36.696856 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:36.696824    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05300884-0a61-4b61-9637-add8ddfff1c5-original-pull-secret\") pod \"global-pull-secret-syncer-zgsll\" (UID: \"05300884-0a61-4b61-9637-add8ddfff1c5\") " pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:33:36.782856 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:36.782828    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q87t4\" (UniqueName: \"kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4\") pod \"network-check-target-lvxg5\" (UID: \"2fc30bdf-7515-45f6-9c88-2f976891430f\") " pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:33:36.786038 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:36.786021    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 19:33:36.795834 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:36.795812    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 19:33:36.802465 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:36.802448    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgsll"
Jul 06 19:33:36.806390 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:36.806370    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q87t4\" (UniqueName: \"kubernetes.io/projected/2fc30bdf-7515-45f6-9c88-2f976891430f-kube-api-access-q87t4\") pod \"network-check-target-lvxg5\" (UID: \"2fc30bdf-7515-45f6-9c88-2f976891430f\") " pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:33:36.813495 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:36.813462    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4gp52\""
Jul 06 19:33:36.820913 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:36.820892    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:33:36.940599 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:36.940540    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zgsll"]
Jul 06 19:33:36.943523 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:33:36.943479    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05300884_0a61_4b61_9637_add8ddfff1c5.slice/crio-b89e1be1c52acfd28e379b18bf8fcd6c27fa4e5286f8082616fa638704ddd3e5 WatchSource:0}: Error finding container b89e1be1c52acfd28e379b18bf8fcd6c27fa4e5286f8082616fa638704ddd3e5: Status 404 returned error can't find the container with id b89e1be1c52acfd28e379b18bf8fcd6c27fa4e5286f8082616fa638704ddd3e5
Jul 06 19:33:36.957176 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:36.957159    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lvxg5"]
Jul 06 19:33:36.959227 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:33:36.959202    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc30bdf_7515_45f6_9c88_2f976891430f.slice/crio-865a7fdc933c8a6f8a5bf2a02e07071cb8530411e3a6c8aaa6232aec7b06cc13 WatchSource:0}: Error finding container 865a7fdc933c8a6f8a5bf2a02e07071cb8530411e3a6c8aaa6232aec7b06cc13: Status 404 returned error can't find the container with id 865a7fdc933c8a6f8a5bf2a02e07071cb8530411e3a6c8aaa6232aec7b06cc13
Jul 06 19:33:37.143055 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:37.143031    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lvxg5" event={"ID":"2fc30bdf-7515-45f6-9c88-2f976891430f","Type":"ContainerStarted","Data":"865a7fdc933c8a6f8a5bf2a02e07071cb8530411e3a6c8aaa6232aec7b06cc13"}
Jul 06 19:33:37.144163 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:37.144141    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zgsll" event={"ID":"05300884-0a61-4b61-9637-add8ddfff1c5","Type":"ContainerStarted","Data":"b89e1be1c52acfd28e379b18bf8fcd6c27fa4e5286f8082616fa638704ddd3e5"}
Jul 06 19:33:39.201182 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:39.201148    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:33:39.201648 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:39.201254    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:33:39.201648 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:39.201366    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 19:33:39.201648 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:39.201383    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7788f56567-99q8x: secret "image-registry-tls" not found
Jul 06 19:33:39.201648 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:39.201441    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls podName:99cfd3ef-94a2-464d-aca2-561f8e30939e nodeName:}" failed. No retries permitted until 2026-07-06 19:34:11.201421573 +0000 UTC m=+99.742598244 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls") pod "image-registry-7788f56567-99q8x" (UID: "99cfd3ef-94a2-464d-aca2-561f8e30939e") : secret "image-registry-tls" not found
Jul 06 19:33:39.201648 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:39.201367    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 19:33:39.201648 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:33:39.201532    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls podName:44bd8f04-80fc-4f99-97f3-3f84666ae1d0 nodeName:}" failed. No retries permitted until 2026-07-06 19:34:11.201509453 +0000 UTC m=+99.742686126 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls") pod "dns-default-phrpj" (UID: "44bd8f04-80fc-4f99-97f3-3f84666ae1d0") : secret "dns-default-metrics-tls" not found
Jul 06 19:33:41.159906 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:41.159866    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lvxg5" event={"ID":"2fc30bdf-7515-45f6-9c88-2f976891430f","Type":"ContainerStarted","Data":"d16df1875b9e7907dcbec8fed6d70a97cb3410c2598e2d14e67083e5e115babe"}
Jul 06 19:33:41.160308 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:41.159962    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:33:41.179784 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:41.179733    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lvxg5" podStartSLOduration=65.770266551 podStartE2EDuration="1m9.179718271s" totalImagesPullingTime="3.40945172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:41.179449177 +0000 UTC m=+69.720625852" watchObservedRunningTime="2026-07-06 19:33:41.179718271 +0000 UTC m=+69.720894945"
Jul 06 19:33:43.168227 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:43.168185    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zgsll" event={"ID":"05300884-0a61-4b61-9637-add8ddfff1c5","Type":"ContainerStarted","Data":"521e8e25bda5ad18db9136cb0d6f2cbe7e7ee6faf8fa57fa6eb7781d63e49d0d"}
Jul 06 19:33:43.187501 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:43.187447    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zgsll" podStartSLOduration=64.776651385 podStartE2EDuration="1m10.187432006s" totalImagesPullingTime="5.410780621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:43.187215674 +0000 UTC m=+71.728392351" watchObservedRunningTime="2026-07-06 19:33:43.187432006 +0000 UTC m=+71.728608682"
Jul 06 19:33:51.721688 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:51.721643    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:33:53.019351 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:33:53.019311    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:11.211242 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:11.211142    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:34:11.211242 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:11.211231    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls\") pod \"image-registry-7788f56567-99q8x\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") " pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:34:11.211688 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:11.211323    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 19:34:11.211688 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:11.211397    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls podName:44bd8f04-80fc-4f99-97f3-3f84666ae1d0 nodeName:}" failed. No retries permitted until 2026-07-06 19:35:15.21137899 +0000 UTC m=+163.752555648 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls") pod "dns-default-phrpj" (UID: "44bd8f04-80fc-4f99-97f3-3f84666ae1d0") : secret "dns-default-metrics-tls" not found
Jul 06 19:34:11.211688 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:11.211326    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 19:34:11.211688 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:11.211433    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7788f56567-99q8x: secret "image-registry-tls" not found
Jul 06 19:34:11.211688 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:11.211512    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls podName:99cfd3ef-94a2-464d-aca2-561f8e30939e nodeName:}" failed. No retries permitted until 2026-07-06 19:35:15.211477802 +0000 UTC m=+163.752654456 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls") pod "image-registry-7788f56567-99q8x" (UID: "99cfd3ef-94a2-464d-aca2-561f8e30939e") : secret "image-registry-tls" not found
Jul 06 19:34:12.165464 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:12.165430    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lvxg5"
Jul 06 19:34:14.664433 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:14.664402    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-z5d9w"]
Jul 06 19:34:14.699148 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:14.699116    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-z5d9w"]
Jul 06 19:34:14.699279 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:14.699206    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-z5d9w"
Jul 06 19:34:14.702537 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:14.702514    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8nlqr\""
Jul 06 19:34:14.736119 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:14.736092    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9hfx\" (UniqueName: \"kubernetes.io/projected/f7a98716-10fc-4a08-a989-db7c5b3eb6c3-kube-api-access-d9hfx\") pod \"network-check-source-989f8b8dc-z5d9w\" (UID: \"f7a98716-10fc-4a08-a989-db7c5b3eb6c3\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-z5d9w"
Jul 06 19:34:14.836400 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:14.836380    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9hfx\" (UniqueName: \"kubernetes.io/projected/f7a98716-10fc-4a08-a989-db7c5b3eb6c3-kube-api-access-d9hfx\") pod \"network-check-source-989f8b8dc-z5d9w\" (UID: \"f7a98716-10fc-4a08-a989-db7c5b3eb6c3\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-z5d9w"
Jul 06 19:34:14.842752 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:14.842718    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:14.846215 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:14.846197    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9hfx\" (UniqueName: \"kubernetes.io/projected/f7a98716-10fc-4a08-a989-db7c5b3eb6c3-kube-api-access-d9hfx\") pod \"network-check-source-989f8b8dc-z5d9w\" (UID: \"f7a98716-10fc-4a08-a989-db7c5b3eb6c3\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-z5d9w"
Jul 06 19:34:14.923176 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:14.923115    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:15.007712 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:15.007690    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-z5d9w"
Jul 06 19:34:15.141204 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:15.141178    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-z5d9w"]
Jul 06 19:34:15.144061 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:34:15.144031    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a98716_10fc_4a08_a989_db7c5b3eb6c3.slice/crio-b1534dd9b60378966db951719d7523cfd2a2ea367a7e8b09946e0e0ef3ac206d WatchSource:0}: Error finding container b1534dd9b60378966db951719d7523cfd2a2ea367a7e8b09946e0e0ef3ac206d: Status 404 returned error can't find the container with id b1534dd9b60378966db951719d7523cfd2a2ea367a7e8b09946e0e0ef3ac206d
Jul 06 19:34:15.267171 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:15.266989    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-z5d9w" event={"ID":"f7a98716-10fc-4a08-a989-db7c5b3eb6c3","Type":"ContainerStarted","Data":"632c4163dc9a5367ba7ab1f125aee208a7b0dd6d31de713140df70747f545f1e"}
Jul 06 19:34:15.267171 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:15.267044    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-z5d9w" event={"ID":"f7a98716-10fc-4a08-a989-db7c5b3eb6c3","Type":"ContainerStarted","Data":"b1534dd9b60378966db951719d7523cfd2a2ea367a7e8b09946e0e0ef3ac206d"}
Jul 06 19:34:15.285975 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:15.285932    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-z5d9w" podStartSLOduration=1.285916877 podStartE2EDuration="1.285916877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:15.284342987 +0000 UTC m=+103.825519655" watchObservedRunningTime="2026-07-06 19:34:15.285916877 +0000 UTC m=+103.827093569"
Jul 06 19:34:17.342953 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:17.342916    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:17.422644 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:17.422608    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:19.424343 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:19.424305    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:19.494625 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:19.494596    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:19.867558 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:19.867532    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:19.937573 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:19.937542    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:20.132246 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.132160    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:20.228566 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.228537    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:20.269569 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.267359    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-426cv"]
Jul 06 19:34:20.273341 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.273314    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:20.276597 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.276575    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 06 19:34:20.276935 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.276916    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 06 19:34:20.277938 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.277915    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 06 19:34:20.277938 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.277930    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-npgh8\""
Jul 06 19:34:20.278075 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.277963    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 06 19:34:20.286152 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.286124    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-426cv"]
Jul 06 19:34:20.327505 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.327451    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:20.374559 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.374537    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n4vn\" (UniqueName: \"kubernetes.io/projected/a71999f1-e3a9-4867-a054-9973c666e32c-kube-api-access-2n4vn\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:20.374636 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.374601    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a71999f1-e3a9-4867-a054-9973c666e32c-data-volume\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:20.374675 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.374639    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a71999f1-e3a9-4867-a054-9973c666e32c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:20.374712 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.374685    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a71999f1-e3a9-4867-a054-9973c666e32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:20.374749 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.374716    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a71999f1-e3a9-4867-a054-9973c666e32c-crio-socket\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:20.399597 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.399534    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:20.475621 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.475595    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4vn\" (UniqueName: \"kubernetes.io/projected/a71999f1-e3a9-4867-a054-9973c666e32c-kube-api-access-2n4vn\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:20.475966 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.475660    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a71999f1-e3a9-4867-a054-9973c666e32c-data-volume\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:20.475966 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.475680    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a71999f1-e3a9-4867-a054-9973c666e32c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:20.475966 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.475704    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a71999f1-e3a9-4867-a054-9973c666e32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:20.475966 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.475730    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a71999f1-e3a9-4867-a054-9973c666e32c-crio-socket\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:20.475966 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:20.475799    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 19:34:20.475966 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.475837    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a71999f1-e3a9-4867-a054-9973c666e32c-crio-socket\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:20.475966 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:20.475878    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a71999f1-e3a9-4867-a054-9973c666e32c-kube-rbac-proxy-cm podName:a71999f1-e3a9-4867-a054-9973c666e32c nodeName:}" failed. No retries permitted until 2026-07-06 19:34:20.975861915 +0000 UTC m=+109.517038571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a71999f1-e3a9-4867-a054-9973c666e32c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-426cv" (UID: "a71999f1-e3a9-4867-a054-9973c666e32c") : configmap "kube-rbac-proxy" not found
Jul 06 19:34:20.475966 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:20.475892    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 19:34:20.476241 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:20.475981    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a71999f1-e3a9-4867-a054-9973c666e32c-insights-runtime-extractor-tls podName:a71999f1-e3a9-4867-a054-9973c666e32c nodeName:}" failed. No retries permitted until 2026-07-06 19:34:20.975959649 +0000 UTC m=+109.517136308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a71999f1-e3a9-4867-a054-9973c666e32c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-426cv" (UID: "a71999f1-e3a9-4867-a054-9973c666e32c") : secret "insights-runtime-extractor-tls" not found
Jul 06 19:34:20.476241 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.476039    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a71999f1-e3a9-4867-a054-9973c666e32c-data-volume\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:20.492656 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.492629    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n4vn\" (UniqueName: \"kubernetes.io/projected/a71999f1-e3a9-4867-a054-9973c666e32c-kube-api-access-2n4vn\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:20.530087 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.530065    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:20.628895 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.628831    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:20.722840 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.722767    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:20.979960 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.979903    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a71999f1-e3a9-4867-a054-9973c666e32c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:20.979960 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:20.979945    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a71999f1-e3a9-4867-a054-9973c666e32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:20.980088 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:20.980024    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 19:34:20.980088 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:20.980058    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 19:34:20.980088 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:20.980086    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a71999f1-e3a9-4867-a054-9973c666e32c-kube-rbac-proxy-cm podName:a71999f1-e3a9-4867-a054-9973c666e32c nodeName:}" failed. No retries permitted until 2026-07-06 19:34:21.980070788 +0000 UTC m=+110.521247442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a71999f1-e3a9-4867-a054-9973c666e32c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-426cv" (UID: "a71999f1-e3a9-4867-a054-9973c666e32c") : configmap "kube-rbac-proxy" not found
Jul 06 19:34:20.980198 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:20.980135    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a71999f1-e3a9-4867-a054-9973c666e32c-insights-runtime-extractor-tls podName:a71999f1-e3a9-4867-a054-9973c666e32c nodeName:}" failed. No retries permitted until 2026-07-06 19:34:21.980119398 +0000 UTC m=+110.521296051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a71999f1-e3a9-4867-a054-9973c666e32c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-426cv" (UID: "a71999f1-e3a9-4867-a054-9973c666e32c") : secret "insights-runtime-extractor-tls" not found
Jul 06 19:34:21.828273 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:21.828240    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:21.987243 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:21.987211    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a71999f1-e3a9-4867-a054-9973c666e32c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:21.987384 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:21.987249    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a71999f1-e3a9-4867-a054-9973c666e32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:21.987384 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:21.987323    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 19:34:21.987384 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:21.987367    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 19:34:21.987541 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:21.987388    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a71999f1-e3a9-4867-a054-9973c666e32c-kube-rbac-proxy-cm podName:a71999f1-e3a9-4867-a054-9973c666e32c nodeName:}" failed. No retries permitted until 2026-07-06 19:34:23.98737202 +0000 UTC m=+112.528548674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a71999f1-e3a9-4867-a054-9973c666e32c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-426cv" (UID: "a71999f1-e3a9-4867-a054-9973c666e32c") : configmap "kube-rbac-proxy" not found
Jul 06 19:34:21.987541 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:21.987419    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a71999f1-e3a9-4867-a054-9973c666e32c-insights-runtime-extractor-tls podName:a71999f1-e3a9-4867-a054-9973c666e32c nodeName:}" failed. No retries permitted until 2026-07-06 19:34:23.987406556 +0000 UTC m=+112.528583209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a71999f1-e3a9-4867-a054-9973c666e32c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-426cv" (UID: "a71999f1-e3a9-4867-a054-9973c666e32c") : secret "insights-runtime-extractor-tls" not found
Jul 06 19:34:22.005044 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:22.005021    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:22.263212 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:22.263191    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n9f6w_5ea2a94f-253e-4bc7-960f-69e371dbbd6a/dns-node-resolver/0.log"
Jul 06 19:34:23.063522 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:23.063475    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2cwdm_75a74486-1003-45bb-9c55-1cbf52ab4090/node-ca/0.log"
Jul 06 19:34:23.338357 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:23.338289    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:23.998627 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:23.998590    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a71999f1-e3a9-4867-a054-9973c666e32c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:23.998627 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:23.998630    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a71999f1-e3a9-4867-a054-9973c666e32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:23.998845 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:23.998714    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 19:34:23.998845 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:23.998775    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 19:34:23.998845 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:23.998790    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a71999f1-e3a9-4867-a054-9973c666e32c-kube-rbac-proxy-cm podName:a71999f1-e3a9-4867-a054-9973c666e32c nodeName:}" failed. No retries permitted until 2026-07-06 19:34:27.998770628 +0000 UTC m=+116.539947290 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a71999f1-e3a9-4867-a054-9973c666e32c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-426cv" (UID: "a71999f1-e3a9-4867-a054-9973c666e32c") : configmap "kube-rbac-proxy" not found
Jul 06 19:34:23.998845 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:23.998816    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a71999f1-e3a9-4867-a054-9973c666e32c-insights-runtime-extractor-tls podName:a71999f1-e3a9-4867-a054-9973c666e32c nodeName:}" failed. No retries permitted until 2026-07-06 19:34:27.99880468 +0000 UTC m=+116.539981388 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a71999f1-e3a9-4867-a054-9973c666e32c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-426cv" (UID: "a71999f1-e3a9-4867-a054-9973c666e32c") : secret "insights-runtime-extractor-tls" not found
Jul 06 19:34:24.262660 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:24.262622    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n6x2q_3d8e171c-5f31-43a4-ad00-034d98242a89/serve-healthcheck-canary/0.log"
Jul 06 19:34:24.741459 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:24.741389    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:24.856138 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:24.856107    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:24.963142 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:24.963107    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:25.058160 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:25.058134    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:25.159311 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:25.159286    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:25.613030 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:25.612992    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:26.218669 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:26.218634    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:26.620097 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:26.620059    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:26.816215 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:26.816189    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:26.921464 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:26.921396    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:27.424849 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:27.424820    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:27.823449 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:27.823423    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:28.025999 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:28.025971    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a71999f1-e3a9-4867-a054-9973c666e32c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:28.026131 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:28.026004    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a71999f1-e3a9-4867-a054-9973c666e32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:28.026131 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:28.026088    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 19:34:28.026131 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:28.026122    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 19:34:28.026264 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:28.026145    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a71999f1-e3a9-4867-a054-9973c666e32c-kube-rbac-proxy-cm podName:a71999f1-e3a9-4867-a054-9973c666e32c nodeName:}" failed. No retries permitted until 2026-07-06 19:34:36.026130489 +0000 UTC m=+124.567307144 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a71999f1-e3a9-4867-a054-9973c666e32c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-426cv" (UID: "a71999f1-e3a9-4867-a054-9973c666e32c") : configmap "kube-rbac-proxy" not found
Jul 06 19:34:28.026264 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:28.026189    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a71999f1-e3a9-4867-a054-9973c666e32c-insights-runtime-extractor-tls podName:a71999f1-e3a9-4867-a054-9973c666e32c nodeName:}" failed. No retries permitted until 2026-07-06 19:34:36.026172083 +0000 UTC m=+124.567348756 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a71999f1-e3a9-4867-a054-9973c666e32c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-426cv" (UID: "a71999f1-e3a9-4867-a054-9973c666e32c") : secret "insights-runtime-extractor-tls" not found
Jul 06 19:34:28.221536 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:28.221453    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:28.431922 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:28.431889    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:28.618999 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:28.618963    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:28.811271 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:28.811246    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:29.416908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:29.416876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:29.522149 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:29.522121    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:30.014047 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:30.014018    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:30.210719 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:30.210693    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:30.312126 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:30.312047    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:30.423672 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:30.423643    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:30.812887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:30.812858    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:31.457238 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:31.457201    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:31.815910 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:31.815878    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:31.916663 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:31.916629    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:32.213079 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:32.213008    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:32.416731 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:32.416697    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:32.819523 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:32.819472    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:33.106775 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:33.106697    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:33.417701 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:33.417627    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:33.821501 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:33.821452    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:34.018616 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:34.018577    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:34.479838 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:34.479806    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:34.625644 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:34.625612    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:34.822161 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:34.822131    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:35.215081 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:35.214961    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:35.411808 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:35.411775    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:36.084889 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:36.084861    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a71999f1-e3a9-4867-a054-9973c666e32c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:36.085241 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:36.084894    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a71999f1-e3a9-4867-a054-9973c666e32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:36.085459 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:36.085435    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a71999f1-e3a9-4867-a054-9973c666e32c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:36.087359 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:36.087333    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a71999f1-e3a9-4867-a054-9973c666e32c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-426cv\" (UID: \"a71999f1-e3a9-4867-a054-9973c666e32c\") " pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:36.187696 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:36.187673    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-npgh8\""
Jul 06 19:34:36.195318 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:36.195299    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-426cv"
Jul 06 19:34:36.325927 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:36.325905    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-426cv"]
Jul 06 19:34:36.328383 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:34:36.328358    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71999f1_e3a9_4867_a054_9973c666e32c.slice/crio-b9f44ebb7f775715e9d8e527b6d200afa370d6caf0fd3f5508a9952ee5b4db97 WatchSource:0}: Error finding container b9f44ebb7f775715e9d8e527b6d200afa370d6caf0fd3f5508a9952ee5b4db97: Status 404 returned error can't find the container with id b9f44ebb7f775715e9d8e527b6d200afa370d6caf0fd3f5508a9952ee5b4db97
Jul 06 19:34:37.332636 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:37.332598    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-426cv" event={"ID":"a71999f1-e3a9-4867-a054-9973c666e32c","Type":"ContainerStarted","Data":"b9f44ebb7f775715e9d8e527b6d200afa370d6caf0fd3f5508a9952ee5b4db97"}
Jul 06 19:34:38.892279 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:38.892243    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:38.996103 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:38.996072    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:39.093856 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:39.093818    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:39.208014 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:39.207931    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:39.574692 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:39.574662    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:39.642705 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:39.642673    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:39.973512 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:39.973471    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:40.074551 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:40.074522    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:40.347850 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:40.343931    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-426cv" event={"ID":"a71999f1-e3a9-4867-a054-9973c666e32c","Type":"ContainerStarted","Data":"5ce5f3f5fb0aff9cbfb25abbcce646bb0b170adb8faa406f769ff3b9c4ba1ca5"}
Jul 06 19:34:40.377673 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:40.377642    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:40.483456 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:40.483431    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:40.720113 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:40.720091    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs\") pod \"network-metrics-daemon-kbl7t\" (UID: \"2bd4ee01-7358-4e6f-9b67-43d41aef4fc3\") " pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:34:40.722167 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:40.722145    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd4ee01-7358-4e6f-9b67-43d41aef4fc3-metrics-certs\") pod \"network-metrics-daemon-kbl7t\" (UID: \"2bd4ee01-7358-4e6f-9b67-43d41aef4fc3\") " pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:34:40.782925 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:40.782897    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:40.886899 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:40.886876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:41.019668 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:41.019647    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b9cds\""
Jul 06 19:34:41.027248 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:41.027217    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kbl7t"
Jul 06 19:34:41.166019 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:41.165984    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kbl7t"]
Jul 06 19:34:41.168948 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:34:41.168912    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd4ee01_7358_4e6f_9b67_43d41aef4fc3.slice/crio-b7cb540046a32ed5098745a05845ece5b9c56bcb1d8c400dada268f30cbe9e95 WatchSource:0}: Error finding container b7cb540046a32ed5098745a05845ece5b9c56bcb1d8c400dada268f30cbe9e95: Status 404 returned error can't find the container with id b7cb540046a32ed5098745a05845ece5b9c56bcb1d8c400dada268f30cbe9e95
Jul 06 19:34:41.347673 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:41.347601    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kbl7t" event={"ID":"2bd4ee01-7358-4e6f-9b67-43d41aef4fc3","Type":"ContainerStarted","Data":"b7cb540046a32ed5098745a05845ece5b9c56bcb1d8c400dada268f30cbe9e95"}
Jul 06 19:34:41.349567 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:41.349537    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-426cv" event={"ID":"a71999f1-e3a9-4867-a054-9973c666e32c","Type":"ContainerStarted","Data":"7ca119663f33e8bde9e3bace2caa955b6e7c7d73e2792168ee1c29bbba8083c3"}
Jul 06 19:34:41.607997 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:41.607917    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:41.709994 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:41.709928    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:42.355836 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:42.355751    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kbl7t" event={"ID":"2bd4ee01-7358-4e6f-9b67-43d41aef4fc3","Type":"ContainerStarted","Data":"b8c61737805f2c1b3996ac62163d7a5435a16b8502e80ba2ad25212150129a18"}
Jul 06 19:34:42.736076 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:42.735994    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:42.831583 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:42.831541    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:43.360737 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:43.360705    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kbl7t" event={"ID":"2bd4ee01-7358-4e6f-9b67-43d41aef4fc3","Type":"ContainerStarted","Data":"21984d6defd7ffd4c74ebc9b3fe46f9c8b91508ba8bafbce30f1472c267f864a"}
Jul 06 19:34:43.362822 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:43.362800    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-426cv" event={"ID":"a71999f1-e3a9-4867-a054-9973c666e32c","Type":"ContainerStarted","Data":"4dcdd080e31d49217b077daea0963ed5b3c91404d67b11dfecc3e3efb1abb94e"}
Jul 06 19:34:43.379670 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:43.379630    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kbl7t" podStartSLOduration=129.458264677 podStartE2EDuration="2m10.379613787s" totalImagesPullingTime="921.34911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:43.377964879 +0000 UTC m=+131.919141555" watchObservedRunningTime="2026-07-06 19:34:43.379613787 +0000 UTC m=+131.920790463"
Jul 06 19:34:43.397891 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:43.397852    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-426cv" podStartSLOduration=16.603797957 podStartE2EDuration="23.397840538s" totalImagesPullingTime="6.794042581s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:34:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:43.397149287 +0000 UTC m=+131.938325964" watchObservedRunningTime="2026-07-06 19:34:43.397840538 +0000 UTC m=+131.939017213"
Jul 06 19:34:43.478884 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:43.478820    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:43.578983 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:43.578950    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:45.775326 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:45.775286    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:45.839547 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:45.839510    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:49.493792 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:49.493750    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:49.596035 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:49.596007    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:49.642481 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:49.642450    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns" podUID="e79108f0-9a0a-49a4-a720-86af45b7d6bb" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 06 19:34:49.687412 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:49.687386    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:49.806149 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:49.806119    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:50.805014 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:50.804953    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:50.905093 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:50.905054    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:51.740800 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:51.740769    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:51.814963 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:51.814922    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:52.717214 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:52.717148    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:52.819453 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:52.819403    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:52.954152 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:52.954118    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:55.023605 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:55.023578    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:56.376825 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:56.376781    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:56.445338 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:56.445308    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:56.843747 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:56.843714    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:57.060370 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.060342    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:57.138052 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.137985    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7ft7v"]
Jul 06 19:34:57.141195 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.141170    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.144458 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.144438    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 06 19:34:57.144804 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.144782    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 06 19:34:57.144895 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.144794    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 06 19:34:57.144895 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.144844    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 06 19:34:57.145746 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.145729    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 06 19:34:57.145850 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.145736    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4vfng\""
Jul 06 19:34:57.145913 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.145736    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 06 19:34:57.161041 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.161019    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:57.238442 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.238414    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48ceb622-879a-4b7c-a0c7-364c97ad46b8-node-exporter-textfile\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.238442 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.238442    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48ceb622-879a-4b7c-a0c7-364c97ad46b8-node-exporter-wtmp\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.238603 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.238460    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48ceb622-879a-4b7c-a0c7-364c97ad46b8-root\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.238603 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.238481    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48ceb622-879a-4b7c-a0c7-364c97ad46b8-metrics-client-ca\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.238603 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.238576    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48ceb622-879a-4b7c-a0c7-364c97ad46b8-node-exporter-tls\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.238603 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.238602    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48ceb622-879a-4b7c-a0c7-364c97ad46b8-sys\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.238738 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.238619    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48ceb622-879a-4b7c-a0c7-364c97ad46b8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.238738 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.238641    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4f84\" (UniqueName: \"kubernetes.io/projected/48ceb622-879a-4b7c-a0c7-364c97ad46b8-kube-api-access-c4f84\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.238738 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.238680    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48ceb622-879a-4b7c-a0c7-364c97ad46b8-node-exporter-accelerators-collector-config\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.257995 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.257968    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:57.339685 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.339662    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48ceb622-879a-4b7c-a0c7-364c97ad46b8-node-exporter-accelerators-collector-config\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.339786 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.339703    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48ceb622-879a-4b7c-a0c7-364c97ad46b8-node-exporter-textfile\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.339786 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.339720    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48ceb622-879a-4b7c-a0c7-364c97ad46b8-node-exporter-wtmp\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.339786 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.339741    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48ceb622-879a-4b7c-a0c7-364c97ad46b8-root\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.339786 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.339768    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48ceb622-879a-4b7c-a0c7-364c97ad46b8-metrics-client-ca\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.339932 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.339829    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48ceb622-879a-4b7c-a0c7-364c97ad46b8-root\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.339932 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.339866    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48ceb622-879a-4b7c-a0c7-364c97ad46b8-node-exporter-wtmp\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.339932 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.339894    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48ceb622-879a-4b7c-a0c7-364c97ad46b8-node-exporter-tls\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.340062 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.339932    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48ceb622-879a-4b7c-a0c7-364c97ad46b8-sys\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.340062 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.339958    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48ceb622-879a-4b7c-a0c7-364c97ad46b8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.340062 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.339992    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4f84\" (UniqueName: \"kubernetes.io/projected/48ceb622-879a-4b7c-a0c7-364c97ad46b8-kube-api-access-c4f84\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.340062 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:57.340025    2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 06 19:34:57.340062 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.340040    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48ceb622-879a-4b7c-a0c7-364c97ad46b8-sys\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.340253 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:34:57.340091    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48ceb622-879a-4b7c-a0c7-364c97ad46b8-node-exporter-tls podName:48ceb622-879a-4b7c-a0c7-364c97ad46b8 nodeName:}" failed. No retries permitted until 2026-07-06 19:34:57.840071802 +0000 UTC m=+146.381248456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/48ceb622-879a-4b7c-a0c7-364c97ad46b8-node-exporter-tls") pod "node-exporter-7ft7v" (UID: "48ceb622-879a-4b7c-a0c7-364c97ad46b8") : secret "node-exporter-tls" not found
Jul 06 19:34:57.340253 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.340164    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48ceb622-879a-4b7c-a0c7-364c97ad46b8-node-exporter-textfile\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.340364 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.340343    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48ceb622-879a-4b7c-a0c7-364c97ad46b8-node-exporter-accelerators-collector-config\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.340409 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.340395    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48ceb622-879a-4b7c-a0c7-364c97ad46b8-metrics-client-ca\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.342247 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.342214    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48ceb622-879a-4b7c-a0c7-364c97ad46b8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.353624 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.353598    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4f84\" (UniqueName: \"kubernetes.io/projected/48ceb622-879a-4b7c-a0c7-364c97ad46b8-kube-api-access-c4f84\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.359063 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.359037    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:57.458561 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.458508    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:57.561358 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.561338    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:57.663527 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.663479    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:57.844627 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.844598    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48ceb622-879a-4b7c-a0c7-364c97ad46b8-node-exporter-tls\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:57.846925 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:57.846897    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48ceb622-879a-4b7c-a0c7-364c97ad46b8-node-exporter-tls\") pod \"node-exporter-7ft7v\" (UID: \"48ceb622-879a-4b7c-a0c7-364c97ad46b8\") " pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:58.050987 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:58.050954    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7ft7v"
Jul 06 19:34:58.067085 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:34:58.067055    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ceb622_879a_4b7c_a0c7_364c97ad46b8.slice/crio-c9cb72ff8575094d6e33f5d1ef2c6591fd90d5befca12e07a7840ee08bea949e WatchSource:0}: Error finding container c9cb72ff8575094d6e33f5d1ef2c6591fd90d5befca12e07a7840ee08bea949e: Status 404 returned error can't find the container with id c9cb72ff8575094d6e33f5d1ef2c6591fd90d5befca12e07a7840ee08bea949e
Jul 06 19:34:58.222889 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:58.222831    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:58.337090 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:58.337064    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:58.412446 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:58.412416    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7ft7v" event={"ID":"48ceb622-879a-4b7c-a0c7-364c97ad46b8","Type":"ContainerStarted","Data":"c9cb72ff8575094d6e33f5d1ef2c6591fd90d5befca12e07a7840ee08bea949e"}
Jul 06 19:34:58.664957 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:58.664924    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:34:59.417532 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:59.417474    2562 generic.go:356] "Generic (PLEG): container finished" podID="48ceb622-879a-4b7c-a0c7-364c97ad46b8" containerID="9989270a0a34c46ca7ac1dffec991ab3c273ee3cbbf16815b6b7ce04509fffe0" exitCode=0
Jul 06 19:34:59.417686 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:59.417556    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7ft7v" event={"ID":"48ceb622-879a-4b7c-a0c7-364c97ad46b8","Type":"ContainerDied","Data":"9989270a0a34c46ca7ac1dffec991ab3c273ee3cbbf16815b6b7ce04509fffe0"}
Jul 06 19:34:59.642200 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:34:59.642167    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns" podUID="e79108f0-9a0a-49a4-a720-86af45b7d6bb" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 06 19:35:00.057599 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:00.057564    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:00.153316 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:00.153283    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:00.280560 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:00.280535    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:00.387743 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:00.387677    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:00.423781 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:00.423757    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7ft7v" event={"ID":"48ceb622-879a-4b7c-a0c7-364c97ad46b8","Type":"ContainerStarted","Data":"84b386a18d4023760517de005131fb4de9d27ee358fbf550905da9a702bc7b5a"}
Jul 06 19:35:00.423914 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:00.423814    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7ft7v" event={"ID":"48ceb622-879a-4b7c-a0c7-364c97ad46b8","Type":"ContainerStarted","Data":"b3cd2e6fb0694063cd26ae7caca83249e6371264fe5de611ad457e3ea9f44861"}
Jul 06 19:35:00.458616 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:00.458579    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7ft7v" podStartSLOduration=2.476980193 podStartE2EDuration="3.458566318s" totalImagesPullingTime="920.48937ms" totalInitContainerRuntime="61.096755ms" isStatefulPod=true podCreationTimestamp="2026-07-06 19:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:35:00.456152877 +0000 UTC m=+148.997329564" watchObservedRunningTime="2026-07-06 19:35:00.458566318 +0000 UTC m=+148.999742994"
Jul 06 19:35:00.722883 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:00.722824    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:01.009236 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:01.009204    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:01.885532 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:01.885500    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:02.081123 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:02.081076    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:02.283242 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:02.283210    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:03.590399 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:03.590368    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:04.573415 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:04.573378    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:04.673302 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:04.673268    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:04.893956 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:04.893859    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7788f56567-99q8x"]
Jul 06 19:35:04.894088 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:35:04.894047    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7788f56567-99q8x" podUID="99cfd3ef-94a2-464d-aca2-561f8e30939e"
Jul 06 19:35:05.000883 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.000841    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:05.100953 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.100919    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:05.438215 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.438182    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:35:05.465078 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.465050    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:05.558113 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.558089    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:35:05.587171 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.587137    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:05.704361 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.704297    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99cfd3ef-94a2-464d-aca2-561f8e30939e-trusted-ca\") pod \"99cfd3ef-94a2-464d-aca2-561f8e30939e\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") "
Jul 06 19:35:05.704361 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.704330    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-bound-sa-token\") pod \"99cfd3ef-94a2-464d-aca2-561f8e30939e\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") "
Jul 06 19:35:05.704809 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.704385    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99cfd3ef-94a2-464d-aca2-561f8e30939e-ca-trust-extracted\") pod \"99cfd3ef-94a2-464d-aca2-561f8e30939e\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") "
Jul 06 19:35:05.704809 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.704416    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99cfd3ef-94a2-464d-aca2-561f8e30939e-image-registry-private-configuration\") pod \"99cfd3ef-94a2-464d-aca2-561f8e30939e\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") "
Jul 06 19:35:05.704809 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.704446    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99cfd3ef-94a2-464d-aca2-561f8e30939e-installation-pull-secrets\") pod \"99cfd3ef-94a2-464d-aca2-561f8e30939e\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") "
Jul 06 19:35:05.704809 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.704473    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-certificates\") pod \"99cfd3ef-94a2-464d-aca2-561f8e30939e\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") "
Jul 06 19:35:05.704809 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.704521    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnfbc\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-kube-api-access-bnfbc\") pod \"99cfd3ef-94a2-464d-aca2-561f8e30939e\" (UID: \"99cfd3ef-94a2-464d-aca2-561f8e30939e\") "
Jul 06 19:35:05.704809 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.704741    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99cfd3ef-94a2-464d-aca2-561f8e30939e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "99cfd3ef-94a2-464d-aca2-561f8e30939e" (UID: "99cfd3ef-94a2-464d-aca2-561f8e30939e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 19:35:05.704809 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.704758    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99cfd3ef-94a2-464d-aca2-561f8e30939e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "99cfd3ef-94a2-464d-aca2-561f8e30939e" (UID: "99cfd3ef-94a2-464d-aca2-561f8e30939e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 19:35:05.705186 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.705159    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "99cfd3ef-94a2-464d-aca2-561f8e30939e" (UID: "99cfd3ef-94a2-464d-aca2-561f8e30939e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 19:35:05.706802 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.706773    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99cfd3ef-94a2-464d-aca2-561f8e30939e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "99cfd3ef-94a2-464d-aca2-561f8e30939e" (UID: "99cfd3ef-94a2-464d-aca2-561f8e30939e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:35:05.706909 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.706851    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99cfd3ef-94a2-464d-aca2-561f8e30939e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "99cfd3ef-94a2-464d-aca2-561f8e30939e" (UID: "99cfd3ef-94a2-464d-aca2-561f8e30939e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:35:05.706999 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.706975    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-kube-api-access-bnfbc" (OuterVolumeSpecName: "kube-api-access-bnfbc") pod "99cfd3ef-94a2-464d-aca2-561f8e30939e" (UID: "99cfd3ef-94a2-464d-aca2-561f8e30939e"). InnerVolumeSpecName "kube-api-access-bnfbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:35:05.707052 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.706988    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "99cfd3ef-94a2-464d-aca2-561f8e30939e" (UID: "99cfd3ef-94a2-464d-aca2-561f8e30939e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:35:05.805023 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.804997    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-bound-sa-token\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 19:35:05.805023 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.805019    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/99cfd3ef-94a2-464d-aca2-561f8e30939e-ca-trust-extracted\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 19:35:05.805164 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.805028    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/99cfd3ef-94a2-464d-aca2-561f8e30939e-image-registry-private-configuration\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 19:35:05.805164 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.805038    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/99cfd3ef-94a2-464d-aca2-561f8e30939e-installation-pull-secrets\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 19:35:05.805164 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.805048    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-certificates\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 19:35:05.805164 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.805056    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnfbc\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-kube-api-access-bnfbc\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 19:35:05.805164 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:05.805065    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/99cfd3ef-94a2-464d-aca2-561f8e30939e-trusted-ca\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 19:35:06.441619 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:06.441580    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7788f56567-99q8x"
Jul 06 19:35:06.480808 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:06.480753    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7788f56567-99q8x"]
Jul 06 19:35:06.485297 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:06.485266    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7788f56567-99q8x"]
Jul 06 19:35:06.611681 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:06.611653    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/99cfd3ef-94a2-464d-aca2-561f8e30939e-registry-tls\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 19:35:07.893590 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:07.893556    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99cfd3ef-94a2-464d-aca2-561f8e30939e" path="/var/lib/kubelet/pods/99cfd3ef-94a2-464d-aca2-561f8e30939e/volumes"
Jul 06 19:35:08.376275 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:08.376230    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:08.480460 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:08.480404    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:09.642184 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:09.642147    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns" podUID="e79108f0-9a0a-49a4-a720-86af45b7d6bb" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 06 19:35:09.642604 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:09.642228    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns"
Jul 06 19:35:09.642731 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:09.642697    2562 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"d734ea273b821f861d55a3dab5f271465a9254ad186d95981f54d568a22b78c3"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 06 19:35:09.642769 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:09.642752    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns" podUID="e79108f0-9a0a-49a4-a720-86af45b7d6bb" containerName="service-proxy" containerID="cri-o://d734ea273b821f861d55a3dab5f271465a9254ad186d95981f54d568a22b78c3" gracePeriod=30
Jul 06 19:35:10.114733 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:10.114700    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:10.214971 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:10.214936    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:10.404251 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:35:10.404177    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-phrpj" podUID="44bd8f04-80fc-4f99-97f3-3f84666ae1d0"
Jul 06 19:35:10.457619 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:10.457593    2562 generic.go:356] "Generic (PLEG): container finished" podID="e79108f0-9a0a-49a4-a720-86af45b7d6bb" containerID="d734ea273b821f861d55a3dab5f271465a9254ad186d95981f54d568a22b78c3" exitCode=2
Jul 06 19:35:10.457753 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:10.457665    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-phrpj"
Jul 06 19:35:10.457753 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:10.457675    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns" event={"ID":"e79108f0-9a0a-49a4-a720-86af45b7d6bb","Type":"ContainerDied","Data":"d734ea273b821f861d55a3dab5f271465a9254ad186d95981f54d568a22b78c3"}
Jul 06 19:35:10.457753 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:10.457745    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59c4585f7f-9lnns" event={"ID":"e79108f0-9a0a-49a4-a720-86af45b7d6bb","Type":"ContainerStarted","Data":"837618c51a3a2d93c8931871d33c2c4458307ec8f8cae9290d80c93ecb36d8c2"}
Jul 06 19:35:11.199683 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:11.199603    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:11.292831 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:11.292802    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:11.373941 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:11.373905    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:12.731628 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:12.731589    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:12.794497 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:12.794462    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:13.652625 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:13.652596    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:13.753870 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:13.753831    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:13.818835 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:13.818794    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:15.272374 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:15.272338    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:35:15.274709 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:15.274689    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/44bd8f04-80fc-4f99-97f3-3f84666ae1d0-metrics-tls\") pod \"dns-default-phrpj\" (UID: \"44bd8f04-80fc-4f99-97f3-3f84666ae1d0\") " pod="openshift-dns/dns-default-phrpj"
Jul 06 19:35:15.562124 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:15.562051    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zq5lq\""
Jul 06 19:35:15.569629 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:15.569611    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-phrpj"
Jul 06 19:35:15.700433 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:15.700359    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-phrpj"]
Jul 06 19:35:15.702755 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:35:15.702724    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44bd8f04_80fc_4f99_97f3_3f84666ae1d0.slice/crio-b99d286c07b51b8cff410d2f44da6f6df6a240627b06892370eee3a6fe276915 WatchSource:0}: Error finding container b99d286c07b51b8cff410d2f44da6f6df6a240627b06892370eee3a6fe276915: Status 404 returned error can't find the container with id b99d286c07b51b8cff410d2f44da6f6df6a240627b06892370eee3a6fe276915
Jul 06 19:35:15.808095 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:15.808061    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:15.908506 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:15.908430    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:16.477755 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:16.477718    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-phrpj" event={"ID":"44bd8f04-80fc-4f99-97f3-3f84666ae1d0","Type":"ContainerStarted","Data":"b99d286c07b51b8cff410d2f44da6f6df6a240627b06892370eee3a6fe276915"}
Jul 06 19:35:18.165272 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:18.165203    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:18.227666 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:18.227630    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:18.487204 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:18.487126    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-phrpj" event={"ID":"44bd8f04-80fc-4f99-97f3-3f84666ae1d0","Type":"ContainerStarted","Data":"69737603eeee2c277d242bb04dd9c1f18ff0368676eff36d4b59b7663e5304af"}
Jul 06 19:35:18.487204 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:18.487185    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-phrpj" event={"ID":"44bd8f04-80fc-4f99-97f3-3f84666ae1d0","Type":"ContainerStarted","Data":"73b0d26e0a1b69e1b7fa0b20c03e4be53893dd8bdff86bff5ca21fc962051530"}
Jul 06 19:35:18.487401 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:18.487257    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-phrpj"
Jul 06 19:35:22.475301 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:22.475250    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:22.574294 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:22.574255    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:22.666218 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:22.666184    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:28.493535 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:28.493427    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-phrpj"
Jul 06 19:35:28.523694 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:28.521338    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-phrpj" podStartSLOduration=139.378266072 podStartE2EDuration="2m21.521317331s" totalImagesPullingTime="2.143051259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:35:18.523352721 +0000 UTC m=+167.064529398" watchObservedRunningTime="2026-07-06 19:35:28.521317331 +0000 UTC m=+177.062494008"
Jul 06 19:35:30.786457 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:30.786413    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:30.883545 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:30.883502    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:37.698477 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:37.698434    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:37.855866 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:37.855824    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:45.784934 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:45.784885    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:35:45.850348 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:35:45.850313    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:36:16.623675 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:36:16.623637    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:36:16.918770 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:36:16.918671    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:36:17.019776 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:36:17.019746    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:36:25.652236 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:36:25.652185    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:36:37.087155 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:36:37.087106    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:36:37.188172 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:36:37.188136    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:36:37.289351 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:36:37.289315    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:37:20.590608 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:37:20.590561    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:37:20.653030 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:37:20.653002    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:37:31.811571 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:37:31.811539    2562 kubelet.go:1648] "Image garbage collection succeeded"
Jul 06 19:37:31.923251 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:37:31.923110    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:37:32.005581 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:37:32.005548    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:37:35.610688 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:37:35.610650    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:37:39.008635 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:37:39.008585    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:37:39.107346 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:37:39.107311    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:37:39.181272 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:37:39.181237    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:37:39.941231 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:37:39.941190    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:37:40.014277 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:37:40.014221    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:37:42.141036 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:37:42.140981    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:37:43.860259 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:37:43.860215    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:37:43.927450 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:37:43.927421    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:37:50.306514 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:37:50.306446    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:37:50.406104 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:37:50.406073    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:37:50.505967 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:37:50.505935    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:38:00.796513 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:38:00.796450    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:38:00.890456 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:38:00.890412    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:38:15.649833 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:38:15.649805    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:38:22.268352 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:38:22.268303    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:38:22.369004 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:38:22.368966    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:38:49.392992 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:38:49.392933    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:38:50.251547 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:38:50.251495    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:38:53.002680 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:38:53.002639    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:38:54.698571 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:38:54.698534    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:38:54.835529 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:38:54.835499    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:38:54.960399 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:38:54.960328    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:38:55.146733 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:38:55.146700    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:38:55.259280 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:38:55.259243    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:38:57.524413 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:38:57.524376    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:38:57.729409 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:38:57.729360    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:00.002436 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:00.002398    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:00.101362 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:00.101327    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:05.214098 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:05.214062    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:10.203530 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:10.203468    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:10.303477 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:10.303448    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:11.769745 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:11.769704    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:11.851361 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:11.851328    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:12.259186 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:12.259158    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s7bn8"]
Jul 06 19:39:12.262186 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:12.262167    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s7bn8"
Jul 06 19:39:12.265267 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:12.265249    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 06 19:39:12.267051 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:12.267033    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 06 19:39:12.267153 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:12.267120    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-rsnh9\""
Jul 06 19:39:12.272379 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:12.272359    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s7bn8"]
Jul 06 19:39:12.324257 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:12.324239    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6be9df0f-77fa-4213-ab8a-701bb246bf28-tmp\") pod \"openshift-lws-operator-fd8ccff4c-s7bn8\" (UID: \"6be9df0f-77fa-4213-ab8a-701bb246bf28\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s7bn8"
Jul 06 19:39:12.324352 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:12.324267    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t74cn\" (UniqueName: \"kubernetes.io/projected/6be9df0f-77fa-4213-ab8a-701bb246bf28-kube-api-access-t74cn\") pod \"openshift-lws-operator-fd8ccff4c-s7bn8\" (UID: \"6be9df0f-77fa-4213-ab8a-701bb246bf28\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s7bn8"
Jul 06 19:39:12.425551 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:12.425526    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6be9df0f-77fa-4213-ab8a-701bb246bf28-tmp\") pod \"openshift-lws-operator-fd8ccff4c-s7bn8\" (UID: \"6be9df0f-77fa-4213-ab8a-701bb246bf28\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s7bn8"
Jul 06 19:39:12.425637 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:12.425555    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t74cn\" (UniqueName: \"kubernetes.io/projected/6be9df0f-77fa-4213-ab8a-701bb246bf28-kube-api-access-t74cn\") pod \"openshift-lws-operator-fd8ccff4c-s7bn8\" (UID: \"6be9df0f-77fa-4213-ab8a-701bb246bf28\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s7bn8"
Jul 06 19:39:12.425862 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:12.425846    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6be9df0f-77fa-4213-ab8a-701bb246bf28-tmp\") pod \"openshift-lws-operator-fd8ccff4c-s7bn8\" (UID: \"6be9df0f-77fa-4213-ab8a-701bb246bf28\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s7bn8"
Jul 06 19:39:12.435710 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:12.435685    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t74cn\" (UniqueName: \"kubernetes.io/projected/6be9df0f-77fa-4213-ab8a-701bb246bf28-kube-api-access-t74cn\") pod \"openshift-lws-operator-fd8ccff4c-s7bn8\" (UID: \"6be9df0f-77fa-4213-ab8a-701bb246bf28\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s7bn8"
Jul 06 19:39:12.571234 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:12.571171    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s7bn8"
Jul 06 19:39:12.697708 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:12.697681    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s7bn8"]
Jul 06 19:39:12.703735 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:39:12.703709    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6be9df0f_77fa_4213_ab8a_701bb246bf28.slice/crio-5b2973430063c6c32c55476b1d6f9605f22915d1a6877957c1b6f2fdbcdd9480 WatchSource:0}: Error finding container 5b2973430063c6c32c55476b1d6f9605f22915d1a6877957c1b6f2fdbcdd9480: Status 404 returned error can't find the container with id 5b2973430063c6c32c55476b1d6f9605f22915d1a6877957c1b6f2fdbcdd9480
Jul 06 19:39:12.705203 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:12.705184    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 19:39:13.289634 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:13.289574    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s7bn8" event={"ID":"6be9df0f-77fa-4213-ab8a-701bb246bf28","Type":"ContainerStarted","Data":"5b2973430063c6c32c55476b1d6f9605f22915d1a6877957c1b6f2fdbcdd9480"}
Jul 06 19:39:15.298699 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:15.298667    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s7bn8" event={"ID":"6be9df0f-77fa-4213-ab8a-701bb246bf28","Type":"ContainerStarted","Data":"1954f33fd18a9b3d50e878c707ac1358cf82c8dd892d1ab7ba243e13d41fe111"}
Jul 06 19:39:15.324420 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:15.324376    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s7bn8" podStartSLOduration=1.196233348 podStartE2EDuration="3.324363145s" totalImagesPullingTime="2.128129797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:39:15.3235307 +0000 UTC m=+403.864707373" watchObservedRunningTime="2026-07-06 19:39:15.324363145 +0000 UTC m=+403.865539818"
Jul 06 19:39:16.099849 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:16.099811    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:16.199367 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:16.199335    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:17.833833 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:17.833801    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:31.091788 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:31.091750    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:31.206965 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:31.206912    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:31.665805 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:31.665764    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:32.045792 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.045760    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:32.150463 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.150424    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:32.177340 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.177310    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"]
Jul 06 19:39:32.180398 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.180381    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"
Jul 06 19:39:32.184131 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.184108    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 06 19:39:32.184131 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.184126    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jxxt7\""
Jul 06 19:39:32.184286 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.184115    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 06 19:39:32.184286 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.184168    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 06 19:39:32.184526 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.184511    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 06 19:39:32.213266 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.213235    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"]
Jul 06 19:39:32.265873 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.265848    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/744c1aa4-0e00-4bbf-be88-b6c364363c84-tmp\") pod \"opendatahub-operator-controller-manager-f6756874-cnbnl\" (UID: \"744c1aa4-0e00-4bbf-be88-b6c364363c84\") " pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"
Jul 06 19:39:32.266031 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.265879    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/744c1aa4-0e00-4bbf-be88-b6c364363c84-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f6756874-cnbnl\" (UID: \"744c1aa4-0e00-4bbf-be88-b6c364363c84\") " pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"
Jul 06 19:39:32.266031 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.265901    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/744c1aa4-0e00-4bbf-be88-b6c364363c84-webhook-cert\") pod \"opendatahub-operator-controller-manager-f6756874-cnbnl\" (UID: \"744c1aa4-0e00-4bbf-be88-b6c364363c84\") " pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"
Jul 06 19:39:32.266031 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.265935    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/744c1aa4-0e00-4bbf-be88-b6c364363c84-manifests\") pod \"opendatahub-operator-controller-manager-f6756874-cnbnl\" (UID: \"744c1aa4-0e00-4bbf-be88-b6c364363c84\") " pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"
Jul 06 19:39:32.266031 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.265999    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptcmg\" (UniqueName: \"kubernetes.io/projected/744c1aa4-0e00-4bbf-be88-b6c364363c84-kube-api-access-ptcmg\") pod \"opendatahub-operator-controller-manager-f6756874-cnbnl\" (UID: \"744c1aa4-0e00-4bbf-be88-b6c364363c84\") " pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"
Jul 06 19:39:32.329743 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.329651    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:32.366643 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.366620    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/744c1aa4-0e00-4bbf-be88-b6c364363c84-manifests\") pod \"opendatahub-operator-controller-manager-f6756874-cnbnl\" (UID: \"744c1aa4-0e00-4bbf-be88-b6c364363c84\") " pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"
Jul 06 19:39:32.366763 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.366649    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptcmg\" (UniqueName: \"kubernetes.io/projected/744c1aa4-0e00-4bbf-be88-b6c364363c84-kube-api-access-ptcmg\") pod \"opendatahub-operator-controller-manager-f6756874-cnbnl\" (UID: \"744c1aa4-0e00-4bbf-be88-b6c364363c84\") " pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"
Jul 06 19:39:32.366763 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.366678    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/744c1aa4-0e00-4bbf-be88-b6c364363c84-tmp\") pod \"opendatahub-operator-controller-manager-f6756874-cnbnl\" (UID: \"744c1aa4-0e00-4bbf-be88-b6c364363c84\") " pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"
Jul 06 19:39:32.366876 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.366785    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/744c1aa4-0e00-4bbf-be88-b6c364363c84-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f6756874-cnbnl\" (UID: \"744c1aa4-0e00-4bbf-be88-b6c364363c84\") " pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"
Jul 06 19:39:32.366876 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.366825    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/744c1aa4-0e00-4bbf-be88-b6c364363c84-webhook-cert\") pod \"opendatahub-operator-controller-manager-f6756874-cnbnl\" (UID: \"744c1aa4-0e00-4bbf-be88-b6c364363c84\") " pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"
Jul 06 19:39:32.367076 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.367049    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/744c1aa4-0e00-4bbf-be88-b6c364363c84-manifests\") pod \"opendatahub-operator-controller-manager-f6756874-cnbnl\" (UID: \"744c1aa4-0e00-4bbf-be88-b6c364363c84\") " pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"
Jul 06 19:39:32.369082 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.369055    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/744c1aa4-0e00-4bbf-be88-b6c364363c84-tmp\") pod \"opendatahub-operator-controller-manager-f6756874-cnbnl\" (UID: \"744c1aa4-0e00-4bbf-be88-b6c364363c84\") " pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"
Jul 06 19:39:32.369223 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.369195    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/744c1aa4-0e00-4bbf-be88-b6c364363c84-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f6756874-cnbnl\" (UID: \"744c1aa4-0e00-4bbf-be88-b6c364363c84\") " pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"
Jul 06 19:39:32.369323 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.369307    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/744c1aa4-0e00-4bbf-be88-b6c364363c84-webhook-cert\") pod \"opendatahub-operator-controller-manager-f6756874-cnbnl\" (UID: \"744c1aa4-0e00-4bbf-be88-b6c364363c84\") " pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"
Jul 06 19:39:32.378058 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.378037    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptcmg\" (UniqueName: \"kubernetes.io/projected/744c1aa4-0e00-4bbf-be88-b6c364363c84-kube-api-access-ptcmg\") pod \"opendatahub-operator-controller-manager-f6756874-cnbnl\" (UID: \"744c1aa4-0e00-4bbf-be88-b6c364363c84\") " pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"
Jul 06 19:39:32.431686 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.431653    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:32.490653 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.490630    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"
Jul 06 19:39:32.536432 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.536393    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:32.631739 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.631717    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"]
Jul 06 19:39:32.636178 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:39:32.636147    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744c1aa4_0e00_4bbf_be88_b6c364363c84.slice/crio-fd499ba9a6d92959215e27cf68eb9e9b50641b7e1fe53b6878a74bd546400ee4 WatchSource:0}: Error finding container fd499ba9a6d92959215e27cf68eb9e9b50641b7e1fe53b6878a74bd546400ee4: Status 404 returned error can't find the container with id fd499ba9a6d92959215e27cf68eb9e9b50641b7e1fe53b6878a74bd546400ee4
Jul 06 19:39:32.637410 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.637362    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:32.737324 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.737286    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:32.835506 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:32.835455    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:33.366795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:33.366751    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl" event={"ID":"744c1aa4-0e00-4bbf-be88-b6c364363c84","Type":"ContainerStarted","Data":"fd499ba9a6d92959215e27cf68eb9e9b50641b7e1fe53b6878a74bd546400ee4"}
Jul 06 19:39:36.379569 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:36.379532    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl" event={"ID":"744c1aa4-0e00-4bbf-be88-b6c364363c84","Type":"ContainerStarted","Data":"6b01d0807b795ca16877691cfa4804d09cc92e71dfc8423b4a3df18bfa12244d"}
Jul 06 19:39:37.385602 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:37.385560    2562 generic.go:356] "Generic (PLEG): container finished" podID="744c1aa4-0e00-4bbf-be88-b6c364363c84" containerID="6b01d0807b795ca16877691cfa4804d09cc92e71dfc8423b4a3df18bfa12244d" exitCode=0
Jul 06 19:39:37.386006 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:37.385612    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl" event={"ID":"744c1aa4-0e00-4bbf-be88-b6c364363c84","Type":"ContainerDied","Data":"6b01d0807b795ca16877691cfa4804d09cc92e71dfc8423b4a3df18bfa12244d"}
Jul 06 19:39:38.392104 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:38.392070    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl" event={"ID":"744c1aa4-0e00-4bbf-be88-b6c364363c84","Type":"ContainerStarted","Data":"e9f7338602609d6ae81ea2ab8e214194d500763cb677a9f2c8edffe2166ec28e"}
Jul 06 19:39:38.392525 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:38.392203    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"
Jul 06 19:39:38.421684 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:38.421635    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl" podStartSLOduration=2.25612797 podStartE2EDuration="6.421620371s" totalImagesPullingTime="3.078035543s" totalInitContainerRuntime="1.087456858s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:39:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:39:38.419599355 +0000 UTC m=+426.960776031" watchObservedRunningTime="2026-07-06 19:39:38.421620371 +0000 UTC m=+426.962797048"
Jul 06 19:39:40.032687 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:40.032655    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:40.125858 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:40.125829    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:49.398459 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:49.398420    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-f6756874-cnbnl"
Jul 06 19:39:51.189963 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:51.189929    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:51.353617 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:51.353582    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:52.307319 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:52.307276    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7xggj"]
Jul 06 19:39:52.325633 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:52.325605    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7xggj"]
Jul 06 19:39:52.325795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:52.325718    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-7xggj"
Jul 06 19:39:52.329267 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:52.329239    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-5jrm2\""
Jul 06 19:39:52.329408 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:52.329334    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 06 19:39:52.405790 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:52.405760    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:52.511927 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:52.511879    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:52.515853 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:52.515829    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60dc1c22-9b70-47f5-8e05-81797da863b4-cert\") pod \"odh-model-controller-7998c67dc4-7xggj\" (UID: \"60dc1c22-9b70-47f5-8e05-81797da863b4\") " pod="opendatahub/odh-model-controller-7998c67dc4-7xggj"
Jul 06 19:39:52.515947 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:52.515876    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pc9m\" (UniqueName: \"kubernetes.io/projected/60dc1c22-9b70-47f5-8e05-81797da863b4-kube-api-access-4pc9m\") pod \"odh-model-controller-7998c67dc4-7xggj\" (UID: \"60dc1c22-9b70-47f5-8e05-81797da863b4\") " pod="opendatahub/odh-model-controller-7998c67dc4-7xggj"
Jul 06 19:39:52.616272 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:52.616205    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:52.616272 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:52.616251    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pc9m\" (UniqueName: \"kubernetes.io/projected/60dc1c22-9b70-47f5-8e05-81797da863b4-kube-api-access-4pc9m\") pod \"odh-model-controller-7998c67dc4-7xggj\" (UID: \"60dc1c22-9b70-47f5-8e05-81797da863b4\") " pod="opendatahub/odh-model-controller-7998c67dc4-7xggj"
Jul 06 19:39:52.616412 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:52.616351    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60dc1c22-9b70-47f5-8e05-81797da863b4-cert\") pod \"odh-model-controller-7998c67dc4-7xggj\" (UID: \"60dc1c22-9b70-47f5-8e05-81797da863b4\") " pod="opendatahub/odh-model-controller-7998c67dc4-7xggj"
Jul 06 19:39:52.616458 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:39:52.616450    2562 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 06 19:39:52.616545 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:39:52.616534    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60dc1c22-9b70-47f5-8e05-81797da863b4-cert podName:60dc1c22-9b70-47f5-8e05-81797da863b4 nodeName:}" failed. No retries permitted until 2026-07-06 19:39:53.116518956 +0000 UTC m=+441.657695610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/60dc1c22-9b70-47f5-8e05-81797da863b4-cert") pod "odh-model-controller-7998c67dc4-7xggj" (UID: "60dc1c22-9b70-47f5-8e05-81797da863b4") : secret "odh-model-controller-webhook-cert" not found
Jul 06 19:39:52.633382 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:52.633355    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pc9m\" (UniqueName: \"kubernetes.io/projected/60dc1c22-9b70-47f5-8e05-81797da863b4-kube-api-access-4pc9m\") pod \"odh-model-controller-7998c67dc4-7xggj\" (UID: \"60dc1c22-9b70-47f5-8e05-81797da863b4\") " pod="opendatahub/odh-model-controller-7998c67dc4-7xggj"
Jul 06 19:39:52.775644 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:52.775614    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:53.048558 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:53.048524    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:53.119717 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:53.119690    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60dc1c22-9b70-47f5-8e05-81797da863b4-cert\") pod \"odh-model-controller-7998c67dc4-7xggj\" (UID: \"60dc1c22-9b70-47f5-8e05-81797da863b4\") " pod="opendatahub/odh-model-controller-7998c67dc4-7xggj"
Jul 06 19:39:53.121919 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:53.121897    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60dc1c22-9b70-47f5-8e05-81797da863b4-cert\") pod \"odh-model-controller-7998c67dc4-7xggj\" (UID: \"60dc1c22-9b70-47f5-8e05-81797da863b4\") " pod="opendatahub/odh-model-controller-7998c67dc4-7xggj"
Jul 06 19:39:53.237285 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:53.237249    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-7xggj"
Jul 06 19:39:53.385395 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:53.385371    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7xggj"]
Jul 06 19:39:53.386935 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:39:53.386905    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60dc1c22_9b70_47f5_8e05_81797da863b4.slice/crio-980c8898e584ff8918e1c36edb33991faa5a2da7e326c6f6900ca4ed7c93ef02 WatchSource:0}: Error finding container 980c8898e584ff8918e1c36edb33991faa5a2da7e326c6f6900ca4ed7c93ef02: Status 404 returned error can't find the container with id 980c8898e584ff8918e1c36edb33991faa5a2da7e326c6f6900ca4ed7c93ef02
Jul 06 19:39:53.389366 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:53.389334    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:53.447185 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:53.447161    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-7xggj" event={"ID":"60dc1c22-9b70-47f5-8e05-81797da863b4","Type":"ContainerStarted","Data":"980c8898e584ff8918e1c36edb33991faa5a2da7e326c6f6900ca4ed7c93ef02"}
Jul 06 19:39:54.430389 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:54.430308    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:39:56.460872 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:56.460799    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-7xggj" event={"ID":"60dc1c22-9b70-47f5-8e05-81797da863b4","Type":"ContainerStarted","Data":"ff5f46498aa3e146d336a53c9a3a7133a3d442a36d307e48871727b6af1b44b8"}
Jul 06 19:39:56.461211 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:56.460970    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-7xggj"
Jul 06 19:39:56.501195 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:39:56.501143    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-7xggj" podStartSLOduration=1.720756335 podStartE2EDuration="4.501122948s" totalImagesPullingTime="2.780366613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:39:56.496762839 +0000 UTC m=+445.037939514" watchObservedRunningTime="2026-07-06 19:39:56.501122948 +0000 UTC m=+445.042299624"
Jul 06 19:40:02.770697 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:02.770665    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:02.947789 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:02.947756    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-696fc764fb-fs5fm"]
Jul 06 19:40:02.950587 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:02.950569    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-696fc764fb-fs5fm"
Jul 06 19:40:02.955040 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:40:02.955004    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-200.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-200.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 06 19:40:02.955040 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:40:02.955026    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-auth-proxy-creds\" is forbidden: User \"system:node:ip-10-0-128-200.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-200.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" type="*v1.Secret"
Jul 06 19:40:02.955186 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:40:02.955062    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kube-auth-proxy-696fc764fb-fs5fm\" is forbidden: User \"system:node:ip-10-0-128-200.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-200.ec2.internal' and this object" podUID="f9961d6a-cdc0-40da-a047-8061fb34902f" pod="openshift-ingress/kube-auth-proxy-696fc764fb-fs5fm"
Jul 06 19:40:02.957145 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:40:02.957118    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-auth-proxy-tls\" is forbidden: User \"system:node:ip-10-0-128-200.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-200.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret"
Jul 06 19:40:02.958135 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:02.958117    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 06 19:40:02.989350 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:02.989325    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-696fc764fb-fs5fm"]
Jul 06 19:40:02.990787 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:02.990766    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2qcb\" (UniqueName: \"kubernetes.io/projected/f9961d6a-cdc0-40da-a047-8061fb34902f-kube-api-access-g2qcb\") pod \"kube-auth-proxy-696fc764fb-fs5fm\" (UID: \"f9961d6a-cdc0-40da-a047-8061fb34902f\") " pod="openshift-ingress/kube-auth-proxy-696fc764fb-fs5fm"
Jul 06 19:40:02.990887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:02.990840    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9961d6a-cdc0-40da-a047-8061fb34902f-tmp\") pod \"kube-auth-proxy-696fc764fb-fs5fm\" (UID: \"f9961d6a-cdc0-40da-a047-8061fb34902f\") " pod="openshift-ingress/kube-auth-proxy-696fc764fb-fs5fm"
Jul 06 19:40:02.990887 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:02.990876    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9961d6a-cdc0-40da-a047-8061fb34902f-tls-certs\") pod \"kube-auth-proxy-696fc764fb-fs5fm\" (UID: \"f9961d6a-cdc0-40da-a047-8061fb34902f\") " pod="openshift-ingress/kube-auth-proxy-696fc764fb-fs5fm"
Jul 06 19:40:03.045782 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.045699    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:03.091762 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.091740    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9961d6a-cdc0-40da-a047-8061fb34902f-tmp\") pod \"kube-auth-proxy-696fc764fb-fs5fm\" (UID: \"f9961d6a-cdc0-40da-a047-8061fb34902f\") " pod="openshift-ingress/kube-auth-proxy-696fc764fb-fs5fm"
Jul 06 19:40:03.091875 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.091781    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9961d6a-cdc0-40da-a047-8061fb34902f-tls-certs\") pod \"kube-auth-proxy-696fc764fb-fs5fm\" (UID: \"f9961d6a-cdc0-40da-a047-8061fb34902f\") " pod="openshift-ingress/kube-auth-proxy-696fc764fb-fs5fm"
Jul 06 19:40:03.091875 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.091817    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2qcb\" (UniqueName: \"kubernetes.io/projected/f9961d6a-cdc0-40da-a047-8061fb34902f-kube-api-access-g2qcb\") pod \"kube-auth-proxy-696fc764fb-fs5fm\" (UID: \"f9961d6a-cdc0-40da-a047-8061fb34902f\") " pod="openshift-ingress/kube-auth-proxy-696fc764fb-fs5fm"
Jul 06 19:40:03.093977 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.093952    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9961d6a-cdc0-40da-a047-8061fb34902f-tmp\") pod \"kube-auth-proxy-696fc764fb-fs5fm\" (UID: \"f9961d6a-cdc0-40da-a047-8061fb34902f\") " pod="openshift-ingress/kube-auth-proxy-696fc764fb-fs5fm"
Jul 06 19:40:03.161517 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.161468    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:03.209645 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.209619    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cndcc"]
Jul 06 19:40:03.212680 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.212665    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cndcc"
Jul 06 19:40:03.217704 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.217685    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 06 19:40:03.218823 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.218806    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-9bg2h\""
Jul 06 19:40:03.219212 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.219170    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zt2qz"]
Jul 06 19:40:03.222704 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.222686    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zt2qz"
Jul 06 19:40:03.225902 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.225888    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 06 19:40:03.226011 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.225900    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-pxw7h\""
Jul 06 19:40:03.254747 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.254724    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cndcc"]
Jul 06 19:40:03.258904 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.258886    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zt2qz"]
Jul 06 19:40:03.295305 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.293231    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4500dcb-07fb-45db-aa5d-e698909e80be-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cndcc\" (UID: \"e4500dcb-07fb-45db-aa5d-e698909e80be\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cndcc"
Jul 06 19:40:03.295305 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.293286    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkj5l\" (UniqueName: \"kubernetes.io/projected/f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3-kube-api-access-bkj5l\") pod \"kserve-controller-manager-d6f4b65b8-zt2qz\" (UID: \"f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zt2qz"
Jul 06 19:40:03.295305 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.293327    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3-cert\") pod \"kserve-controller-manager-d6f4b65b8-zt2qz\" (UID: \"f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zt2qz"
Jul 06 19:40:03.295305 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.293367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lnx2\" (UniqueName: \"kubernetes.io/projected/e4500dcb-07fb-45db-aa5d-e698909e80be-kube-api-access-9lnx2\") pod \"llmisvc-controller-manager-777c6bd7d5-cndcc\" (UID: \"e4500dcb-07fb-45db-aa5d-e698909e80be\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cndcc"
Jul 06 19:40:03.296955 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.296889    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:03.393885 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.393859    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lnx2\" (UniqueName: \"kubernetes.io/projected/e4500dcb-07fb-45db-aa5d-e698909e80be-kube-api-access-9lnx2\") pod \"llmisvc-controller-manager-777c6bd7d5-cndcc\" (UID: \"e4500dcb-07fb-45db-aa5d-e698909e80be\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cndcc"
Jul 06 19:40:03.393992 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.393919    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4500dcb-07fb-45db-aa5d-e698909e80be-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cndcc\" (UID: \"e4500dcb-07fb-45db-aa5d-e698909e80be\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cndcc"
Jul 06 19:40:03.393992 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.393969    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkj5l\" (UniqueName: \"kubernetes.io/projected/f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3-kube-api-access-bkj5l\") pod \"kserve-controller-manager-d6f4b65b8-zt2qz\" (UID: \"f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zt2qz"
Jul 06 19:40:03.394101 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:40:03.393996    2562 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 06 19:40:03.394101 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.394015    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3-cert\") pod \"kserve-controller-manager-d6f4b65b8-zt2qz\" (UID: \"f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zt2qz"
Jul 06 19:40:03.394101 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:40:03.394063    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4500dcb-07fb-45db-aa5d-e698909e80be-cert podName:e4500dcb-07fb-45db-aa5d-e698909e80be nodeName:}" failed. No retries permitted until 2026-07-06 19:40:03.894047276 +0000 UTC m=+452.435223949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e4500dcb-07fb-45db-aa5d-e698909e80be-cert") pod "llmisvc-controller-manager-777c6bd7d5-cndcc" (UID: "e4500dcb-07fb-45db-aa5d-e698909e80be") : secret "llmisvc-webhook-server-cert" not found
Jul 06 19:40:03.394258 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:40:03.394120    2562 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 06 19:40:03.394258 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:40:03.394189    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3-cert podName:f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3 nodeName:}" failed. No retries permitted until 2026-07-06 19:40:03.894171086 +0000 UTC m=+452.435347754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3-cert") pod "kserve-controller-manager-d6f4b65b8-zt2qz" (UID: "f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3") : secret "kserve-webhook-server-cert" not found
Jul 06 19:40:03.410807 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.410779    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkj5l\" (UniqueName: \"kubernetes.io/projected/f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3-kube-api-access-bkj5l\") pod \"kserve-controller-manager-d6f4b65b8-zt2qz\" (UID: \"f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zt2qz"
Jul 06 19:40:03.410890 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.410849    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lnx2\" (UniqueName: \"kubernetes.io/projected/e4500dcb-07fb-45db-aa5d-e698909e80be-kube-api-access-9lnx2\") pod \"llmisvc-controller-manager-777c6bd7d5-cndcc\" (UID: \"e4500dcb-07fb-45db-aa5d-e698909e80be\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cndcc"
Jul 06 19:40:03.589575 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.589511    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:03.897243 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.897170    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3-cert\") pod \"kserve-controller-manager-d6f4b65b8-zt2qz\" (UID: \"f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zt2qz"
Jul 06 19:40:03.897727 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:40:03.897305    2562 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 06 19:40:03.897727 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:40:03.897376    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3-cert podName:f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3 nodeName:}" failed. No retries permitted until 2026-07-06 19:40:04.897355358 +0000 UTC m=+453.438532061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3-cert") pod "kserve-controller-manager-d6f4b65b8-zt2qz" (UID: "f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3") : secret "kserve-webhook-server-cert" not found
Jul 06 19:40:03.897727 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.897457    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4500dcb-07fb-45db-aa5d-e698909e80be-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cndcc\" (UID: \"e4500dcb-07fb-45db-aa5d-e698909e80be\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cndcc"
Jul 06 19:40:03.899953 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.899926    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4500dcb-07fb-45db-aa5d-e698909e80be-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cndcc\" (UID: \"e4500dcb-07fb-45db-aa5d-e698909e80be\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cndcc"
Jul 06 19:40:03.937412 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.937385    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 06 19:40:03.944415 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:03.944384    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9961d6a-cdc0-40da-a047-8061fb34902f-tls-certs\") pod \"kube-auth-proxy-696fc764fb-fs5fm\" (UID: \"f9961d6a-cdc0-40da-a047-8061fb34902f\") " pod="openshift-ingress/kube-auth-proxy-696fc764fb-fs5fm"
Jul 06 19:40:04.049933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:04.049910    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:04.070547 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:04.070523    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 06 19:40:04.079015 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:04.078994    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2qcb\" (UniqueName: \"kubernetes.io/projected/f9961d6a-cdc0-40da-a047-8061fb34902f-kube-api-access-g2qcb\") pod \"kube-auth-proxy-696fc764fb-fs5fm\" (UID: \"f9961d6a-cdc0-40da-a047-8061fb34902f\") " pod="openshift-ingress/kube-auth-proxy-696fc764fb-fs5fm"
Jul 06 19:40:04.122709 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:04.122691    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cndcc"
Jul 06 19:40:04.161519 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:04.161439    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-696fc764fb-fs5fm"
Jul 06 19:40:04.276366 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:04.276343    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cndcc"]
Jul 06 19:40:04.278606 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:40:04.278576    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode4500dcb_07fb_45db_aa5d_e698909e80be.slice/crio-57f5ad22035fc98e860303bd84db3b355e3121f4b6a7a7ae6fa2c12440bb4fb0 WatchSource:0}: Error finding container 57f5ad22035fc98e860303bd84db3b355e3121f4b6a7a7ae6fa2c12440bb4fb0: Status 404 returned error can't find the container with id 57f5ad22035fc98e860303bd84db3b355e3121f4b6a7a7ae6fa2c12440bb4fb0
Jul 06 19:40:04.301933 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:04.301914    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-696fc764fb-fs5fm"]
Jul 06 19:40:04.303930 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:40:04.303906    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9961d6a_cdc0_40da_a047_8061fb34902f.slice/crio-67b14607b0ab94d00838badd4cbaf0031b7a0d8b50e83e787ce9943d6cfb62ef WatchSource:0}: Error finding container 67b14607b0ab94d00838badd4cbaf0031b7a0d8b50e83e787ce9943d6cfb62ef: Status 404 returned error can't find the container with id 67b14607b0ab94d00838badd4cbaf0031b7a0d8b50e83e787ce9943d6cfb62ef
Jul 06 19:40:04.402121 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:04.402099    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 06 19:40:04.467336 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:04.467280    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:04.490275 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:04.490245    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cndcc" event={"ID":"e4500dcb-07fb-45db-aa5d-e698909e80be","Type":"ContainerStarted","Data":"57f5ad22035fc98e860303bd84db3b355e3121f4b6a7a7ae6fa2c12440bb4fb0"}
Jul 06 19:40:04.491199 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:04.491179    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-696fc764fb-fs5fm" event={"ID":"f9961d6a-cdc0-40da-a047-8061fb34902f","Type":"ContainerStarted","Data":"67b14607b0ab94d00838badd4cbaf0031b7a0d8b50e83e787ce9943d6cfb62ef"}
Jul 06 19:40:04.584258 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:04.584239    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:04.681114 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:04.681078    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:04.906872 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:04.906834    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3-cert\") pod \"kserve-controller-manager-d6f4b65b8-zt2qz\" (UID: \"f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zt2qz"
Jul 06 19:40:04.909538 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:04.909513    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3-cert\") pod \"kserve-controller-manager-d6f4b65b8-zt2qz\" (UID: \"f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zt2qz"
Jul 06 19:40:05.031276 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.031243    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zt2qz"
Jul 06 19:40:05.430214 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.430182    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zt2qz"]
Jul 06 19:40:05.448642 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:40:05.448581    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71a2a16_e84f_49dc_bdef_bbeb3c9e67f3.slice/crio-6d11e93d9bca70499bafb31394bdcbe280422efe328d6fc49c4b0a99a67a9f7d WatchSource:0}: Error finding container 6d11e93d9bca70499bafb31394bdcbe280422efe328d6fc49c4b0a99a67a9f7d: Status 404 returned error can't find the container with id 6d11e93d9bca70499bafb31394bdcbe280422efe328d6fc49c4b0a99a67a9f7d
Jul 06 19:40:05.480694 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.480656    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:05.492317 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.492295    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"]
Jul 06 19:40:05.495570 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.495547    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.496757 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.496730    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zt2qz" event={"ID":"f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3","Type":"ContainerStarted","Data":"6d11e93d9bca70499bafb31394bdcbe280422efe328d6fc49c4b0a99a67a9f7d"}
Jul 06 19:40:05.498903 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.498865    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 06 19:40:05.499058 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.498869    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 06 19:40:05.499058 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.499033    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 06 19:40:05.500259 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.500232    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 06 19:40:05.500341 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.500233    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vhtwt\""
Jul 06 19:40:05.512224 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.512198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/341c3a86-c80d-499e-ad5b-24b5bed6ff49-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.512322 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.512249    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/341c3a86-c80d-499e-ad5b-24b5bed6ff49-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.512322 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.512282    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/341c3a86-c80d-499e-ad5b-24b5bed6ff49-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.512322 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.512317    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/341c3a86-c80d-499e-ad5b-24b5bed6ff49-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.512499 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.512355    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/341c3a86-c80d-499e-ad5b-24b5bed6ff49-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.512499 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.512381    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcwc4\" (UniqueName: \"kubernetes.io/projected/341c3a86-c80d-499e-ad5b-24b5bed6ff49-kube-api-access-kcwc4\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.512757 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.512720    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/341c3a86-c80d-499e-ad5b-24b5bed6ff49-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.514107 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.514088    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"]
Jul 06 19:40:05.577967 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.577901    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:05.613613 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.613587    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/341c3a86-c80d-499e-ad5b-24b5bed6ff49-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.613717 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.613629    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/341c3a86-c80d-499e-ad5b-24b5bed6ff49-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.613717 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.613659    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/341c3a86-c80d-499e-ad5b-24b5bed6ff49-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.613717 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.613693    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/341c3a86-c80d-499e-ad5b-24b5bed6ff49-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.613881 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.613730    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/341c3a86-c80d-499e-ad5b-24b5bed6ff49-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.613881 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.613761    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcwc4\" (UniqueName: \"kubernetes.io/projected/341c3a86-c80d-499e-ad5b-24b5bed6ff49-kube-api-access-kcwc4\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.613881 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.613811    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/341c3a86-c80d-499e-ad5b-24b5bed6ff49-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.614462 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.614437    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/341c3a86-c80d-499e-ad5b-24b5bed6ff49-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.616531 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.616512    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/341c3a86-c80d-499e-ad5b-24b5bed6ff49-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.616640 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.616615    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/341c3a86-c80d-499e-ad5b-24b5bed6ff49-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.616853 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.616836    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/341c3a86-c80d-499e-ad5b-24b5bed6ff49-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.616956 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.616936    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/341c3a86-c80d-499e-ad5b-24b5bed6ff49-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.641886 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.641865    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcwc4\" (UniqueName: \"kubernetes.io/projected/341c3a86-c80d-499e-ad5b-24b5bed6ff49-kube-api-access-kcwc4\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.648625 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.648605    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/341c3a86-c80d-499e-ad5b-24b5bed6ff49-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-qg5nc\" (UID: \"341c3a86-c80d-499e-ad5b-24b5bed6ff49\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.807304 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.807272    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:05.966192 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:05.966166    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"]
Jul 06 19:40:05.969048 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:40:05.969025    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341c3a86_c80d_499e_ad5b_24b5bed6ff49.slice/crio-dac391fbc1dd70f6851d80ec8eb8ea1e0be670f82216b97aeb8ce3dd258e5683 WatchSource:0}: Error finding container dac391fbc1dd70f6851d80ec8eb8ea1e0be670f82216b97aeb8ce3dd258e5683: Status 404 returned error can't find the container with id dac391fbc1dd70f6851d80ec8eb8ea1e0be670f82216b97aeb8ce3dd258e5683
Jul 06 19:40:06.503173 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:06.503140    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc" event={"ID":"341c3a86-c80d-499e-ad5b-24b5bed6ff49","Type":"ContainerStarted","Data":"dac391fbc1dd70f6851d80ec8eb8ea1e0be670f82216b97aeb8ce3dd258e5683"}
Jul 06 19:40:07.467774 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:07.467030    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-7xggj"
Jul 06 19:40:07.511333 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:07.511272    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-696fc764fb-fs5fm" event={"ID":"f9961d6a-cdc0-40da-a047-8061fb34902f","Type":"ContainerStarted","Data":"5a83292c7aad09ef1d568a76da04f11b6576218c64607f17014dad4cbcc2ef96"}
Jul 06 19:40:07.565812 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:07.565728    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-696fc764fb-fs5fm" podStartSLOduration=3.042432474 podStartE2EDuration="5.56570752s" totalImagesPullingTime="2.523275046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:40:07.565056091 +0000 UTC m=+456.106232770" watchObservedRunningTime="2026-07-06 19:40:07.56570752 +0000 UTC m=+456.106884200"
Jul 06 19:40:07.821862 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:07.821824    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:09.517202 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:09.517169    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 06 19:40:09.517656 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:09.517242    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 06 19:40:10.527323 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:10.527282    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zt2qz" event={"ID":"f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3","Type":"ContainerStarted","Data":"98707fcff53bd15e73d4db3ac143fd94afcbd7e8df1f9d3d0f74733471942feb"}
Jul 06 19:40:10.527803 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:10.527377    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zt2qz"
Jul 06 19:40:10.529431 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:10.529404    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cndcc" event={"ID":"e4500dcb-07fb-45db-aa5d-e698909e80be","Type":"ContainerStarted","Data":"1be7175f6f792ab106c9bed65db9a2ad0af2afc08b311b9370154aea4f8aaf91"}
Jul 06 19:40:10.529565 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:10.529454    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cndcc"
Jul 06 19:40:10.531188 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:10.531162    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc" event={"ID":"341c3a86-c80d-499e-ad5b-24b5bed6ff49","Type":"ContainerStarted","Data":"6ca84e8b2c4c8fe46080d469cb5a830beaef006d21dd8d829df2e459e41e9718"}
Jul 06 19:40:10.531363 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:10.531344    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:10.532996 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:10.532967    2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-qg5nc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 06 19:40:10.533103 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:10.533021    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc" podUID="341c3a86-c80d-499e-ad5b-24b5bed6ff49" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 19:40:10.573880 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:10.573835    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zt2qz" podStartSLOduration=3.5040600570000002 podStartE2EDuration="7.573819241s" totalImagesPullingTime="4.069759184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:40:10.573692558 +0000 UTC m=+459.114869238" watchObservedRunningTime="2026-07-06 19:40:10.573819241 +0000 UTC m=+459.114995918"
Jul 06 19:40:10.578779 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:10.578752    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:10.612304 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:10.612260    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cndcc" podStartSLOduration=2.382534512 podStartE2EDuration="7.612246868s" totalImagesPullingTime="5.229712356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:40:10.611796624 +0000 UTC m=+459.152973302" watchObservedRunningTime="2026-07-06 19:40:10.612246868 +0000 UTC m=+459.153423544"
Jul 06 19:40:10.651054 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:10.651008    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc" podStartSLOduration=2.1051164 podStartE2EDuration="5.650997328s" totalImagesPullingTime="3.545880928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:40:10.649581709 +0000 UTC m=+459.190758385" watchObservedRunningTime="2026-07-06 19:40:10.650997328 +0000 UTC m=+459.192174004"
Jul 06 19:40:10.690737 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:10.690709    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:11.536742 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:11.536714    2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-qg5nc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 06 19:40:11.537097 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:11.536762    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc" podUID="341c3a86-c80d-499e-ad5b-24b5bed6ff49" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 19:40:11.940216 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:11.940120    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:12.223498 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:12.223403    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:12.447438 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:12.447410    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:14.536978 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:14.536949    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qg5nc"
Jul 06 19:40:41.538700 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:41.538664    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cndcc"
Jul 06 19:40:41.541674 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:41.541651    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zt2qz"
Jul 06 19:40:52.730336 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:52.730294    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:52.835088 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:52.835055    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:52.980195 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:52.980160    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:53.659009 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:53.658962    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:53.868573 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:53.868540    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:58.144497 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:58.144433    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:40:58.212316 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:40:58.212284    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:16.886065 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:16.886020    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:20.144612 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:20.144576    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:22.285061 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:22.285015    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:23.889540 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:23.889431    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:24.292159 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:24.292121    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:24.924149 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:24.924115    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:26.872895 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:26.872855    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-fv64h"]
Jul 06 19:41:26.876236 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:26.876210    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fv64h"
Jul 06 19:41:26.883323 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:26.883303    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-vt6gw\""
Jul 06 19:41:26.883412 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:26.883348    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 06 19:41:26.883502 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:26.883462    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 06 19:41:26.895557 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:26.895533    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t45sp\" (UniqueName: \"kubernetes.io/projected/7ae24278-0051-4c17-be2d-a240ceebc36c-kube-api-access-t45sp\") pod \"authorino-operator-75cc8d5cb7-fv64h\" (UID: \"7ae24278-0051-4c17-be2d-a240ceebc36c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-fv64h"
Jul 06 19:41:26.916646 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:26.916622    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-fv64h"]
Jul 06 19:41:26.996020 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:26.995996    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t45sp\" (UniqueName: \"kubernetes.io/projected/7ae24278-0051-4c17-be2d-a240ceebc36c-kube-api-access-t45sp\") pod \"authorino-operator-75cc8d5cb7-fv64h\" (UID: \"7ae24278-0051-4c17-be2d-a240ceebc36c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-fv64h"
Jul 06 19:41:27.015510 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:27.015467    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t45sp\" (UniqueName: \"kubernetes.io/projected/7ae24278-0051-4c17-be2d-a240ceebc36c-kube-api-access-t45sp\") pod \"authorino-operator-75cc8d5cb7-fv64h\" (UID: \"7ae24278-0051-4c17-be2d-a240ceebc36c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-fv64h"
Jul 06 19:41:27.186778 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:27.186720    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fv64h"
Jul 06 19:41:27.325581 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:27.325411    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-fv64h"]
Jul 06 19:41:27.328590 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:41:27.328552    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae24278_0051_4c17_be2d_a240ceebc36c.slice/crio-4596d008cb77afbd80d23f6d94f659e809dd3867f09cbcdf261b4c7977bc7724 WatchSource:0}: Error finding container 4596d008cb77afbd80d23f6d94f659e809dd3867f09cbcdf261b4c7977bc7724: Status 404 returned error can't find the container with id 4596d008cb77afbd80d23f6d94f659e809dd3867f09cbcdf261b4c7977bc7724
Jul 06 19:41:27.636610 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:27.636584    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:27.775637 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:27.775609    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:27.873366 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:27.873328    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fv64h" event={"ID":"7ae24278-0051-4c17-be2d-a240ceebc36c","Type":"ContainerStarted","Data":"4596d008cb77afbd80d23f6d94f659e809dd3867f09cbcdf261b4c7977bc7724"}
Jul 06 19:41:29.088329 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:29.088292    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:29.883571 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:29.883543    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fv64h" event={"ID":"7ae24278-0051-4c17-be2d-a240ceebc36c","Type":"ContainerStarted","Data":"17d78fd0f6f91ef154188200820cb828494f11736d237bc751471a0fc6891b50"}
Jul 06 19:41:29.883702 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:29.883687    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fv64h"
Jul 06 19:41:29.924040 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:29.923956    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fv64h" podStartSLOduration=1.609018762 podStartE2EDuration="3.923942194s" totalImagesPullingTime="2.314923432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:41:29.921673623 +0000 UTC m=+538.462850299" watchObservedRunningTime="2026-07-06 19:41:29.923942194 +0000 UTC m=+538.465118870"
Jul 06 19:41:30.123134 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:30.123090    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:30.654979 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:30.654942    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:32.200499 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:32.200448    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:32.353273 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:32.353239    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:38.953659 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:38.953614    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:39.064264 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:39.064230    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:40.828287 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:40.828250    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:40.890202 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:40.890175    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fv64h"
Jul 06 19:41:41.139667 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:41.139583    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:45.150524 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:45.150463    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:45.271434 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:45.271396    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:47.635925 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:47.635879    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:47.738470 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:47.738434    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:53.604025 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:53.603986    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:41:53.700063 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:41:53.700016    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:00.735806 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:00.735764    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:00.839403 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:00.839368    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:05.559679 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:05.559635    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:05.657077 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:05.657031    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:05.735351 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:05.735309    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:08.022722 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:08.022674    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:08.129310 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:08.129275    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:11.332955 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:11.332912    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:11.508091 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:11.508060    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:14.289287 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:14.289258    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5wcfv"]
Jul 06 19:42:14.292399 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:14.292383    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5wcfv"
Jul 06 19:42:14.305291 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:14.305273    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 06 19:42:14.311372 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:14.311357    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kztpm\""
Jul 06 19:42:14.323771 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:14.323737    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:14.371197 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:14.371172    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5wcfv"]
Jul 06 19:42:14.416013 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:14.415989    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m97t\" (UniqueName: \"kubernetes.io/projected/f41cd0e8-cba6-4037-8e7d-2af3a82182ac-kube-api-access-9m97t\") pod \"limitador-limitador-64c94fd64f-5wcfv\" (UID: \"f41cd0e8-cba6-4037-8e7d-2af3a82182ac\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5wcfv"
Jul 06 19:42:14.416147 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:14.416131    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f41cd0e8-cba6-4037-8e7d-2af3a82182ac-config-file\") pod \"limitador-limitador-64c94fd64f-5wcfv\" (UID: \"f41cd0e8-cba6-4037-8e7d-2af3a82182ac\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5wcfv"
Jul 06 19:42:14.449787 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:14.449761    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5wcfv"]
Jul 06 19:42:14.492480 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:14.492446    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:14.517322 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:14.517302    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f41cd0e8-cba6-4037-8e7d-2af3a82182ac-config-file\") pod \"limitador-limitador-64c94fd64f-5wcfv\" (UID: \"f41cd0e8-cba6-4037-8e7d-2af3a82182ac\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5wcfv"
Jul 06 19:42:14.517435 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:14.517336    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m97t\" (UniqueName: \"kubernetes.io/projected/f41cd0e8-cba6-4037-8e7d-2af3a82182ac-kube-api-access-9m97t\") pod \"limitador-limitador-64c94fd64f-5wcfv\" (UID: \"f41cd0e8-cba6-4037-8e7d-2af3a82182ac\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5wcfv"
Jul 06 19:42:14.517944 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:14.517923    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f41cd0e8-cba6-4037-8e7d-2af3a82182ac-config-file\") pod \"limitador-limitador-64c94fd64f-5wcfv\" (UID: \"f41cd0e8-cba6-4037-8e7d-2af3a82182ac\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5wcfv"
Jul 06 19:42:14.543203 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:14.543148    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m97t\" (UniqueName: \"kubernetes.io/projected/f41cd0e8-cba6-4037-8e7d-2af3a82182ac-kube-api-access-9m97t\") pod \"limitador-limitador-64c94fd64f-5wcfv\" (UID: \"f41cd0e8-cba6-4037-8e7d-2af3a82182ac\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5wcfv"
Jul 06 19:42:14.602112 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:14.602083    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5wcfv"
Jul 06 19:42:14.749997 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:14.749971    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5wcfv"]
Jul 06 19:42:14.750919 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:42:14.750896    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41cd0e8_cba6_4037_8e7d_2af3a82182ac.slice/crio-551428f1c9badda6a0db86648511ff03f540b737e7366a2d2af91576bf668823 WatchSource:0}: Error finding container 551428f1c9badda6a0db86648511ff03f540b737e7366a2d2af91576bf668823: Status 404 returned error can't find the container with id 551428f1c9badda6a0db86648511ff03f540b737e7366a2d2af91576bf668823
Jul 06 19:42:15.076207 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:15.076176    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5wcfv" event={"ID":"f41cd0e8-cba6-4037-8e7d-2af3a82182ac","Type":"ContainerStarted","Data":"551428f1c9badda6a0db86648511ff03f540b737e7366a2d2af91576bf668823"}
Jul 06 19:42:15.479867 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:15.479622    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:15.623687 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:15.623649    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:18.092141 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:18.092092    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5wcfv" event={"ID":"f41cd0e8-cba6-4037-8e7d-2af3a82182ac","Type":"ContainerStarted","Data":"b64956862ca6dec0c9aaff409ccce66d6cab1857d85da306664b7fd3a343d8cd"}
Jul 06 19:42:18.092667 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:18.092195    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-5wcfv"
Jul 06 19:42:18.143101 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:18.143047    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-5wcfv" podStartSLOduration=1.7381139970000001 podStartE2EDuration="4.143033491s" totalImagesPullingTime="2.404919494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:42:18.141912176 +0000 UTC m=+586.683088884" watchObservedRunningTime="2026-07-06 19:42:18.143033491 +0000 UTC m=+586.684210167"
Jul 06 19:42:29.098055 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:29.098019    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-5wcfv"
Jul 06 19:42:29.622976 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:29.622941    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5wcfv"]
Jul 06 19:42:29.623213 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:29.623182    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-5wcfv" podUID="f41cd0e8-cba6-4037-8e7d-2af3a82182ac" containerName="limitador" containerID="cri-o://b64956862ca6dec0c9aaff409ccce66d6cab1857d85da306664b7fd3a343d8cd" gracePeriod=30
Jul 06 19:42:30.147229 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:30.147200    2562 generic.go:356] "Generic (PLEG): container finished" podID="f41cd0e8-cba6-4037-8e7d-2af3a82182ac" containerID="b64956862ca6dec0c9aaff409ccce66d6cab1857d85da306664b7fd3a343d8cd" exitCode=0
Jul 06 19:42:30.147543 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:30.147270    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5wcfv" event={"ID":"f41cd0e8-cba6-4037-8e7d-2af3a82182ac","Type":"ContainerDied","Data":"b64956862ca6dec0c9aaff409ccce66d6cab1857d85da306664b7fd3a343d8cd"}
Jul 06 19:42:30.165358 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:30.165336    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5wcfv"
Jul 06 19:42:30.229114 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:30.229058    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f41cd0e8-cba6-4037-8e7d-2af3a82182ac-config-file\") pod \"f41cd0e8-cba6-4037-8e7d-2af3a82182ac\" (UID: \"f41cd0e8-cba6-4037-8e7d-2af3a82182ac\") "
Jul 06 19:42:30.229231 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:30.229143    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m97t\" (UniqueName: \"kubernetes.io/projected/f41cd0e8-cba6-4037-8e7d-2af3a82182ac-kube-api-access-9m97t\") pod \"f41cd0e8-cba6-4037-8e7d-2af3a82182ac\" (UID: \"f41cd0e8-cba6-4037-8e7d-2af3a82182ac\") "
Jul 06 19:42:30.229434 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:30.229410    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f41cd0e8-cba6-4037-8e7d-2af3a82182ac-config-file" (OuterVolumeSpecName: "config-file") pod "f41cd0e8-cba6-4037-8e7d-2af3a82182ac" (UID: "f41cd0e8-cba6-4037-8e7d-2af3a82182ac"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 19:42:30.231148 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:30.231128    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41cd0e8-cba6-4037-8e7d-2af3a82182ac-kube-api-access-9m97t" (OuterVolumeSpecName: "kube-api-access-9m97t") pod "f41cd0e8-cba6-4037-8e7d-2af3a82182ac" (UID: "f41cd0e8-cba6-4037-8e7d-2af3a82182ac"). InnerVolumeSpecName "kube-api-access-9m97t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:42:30.330398 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:30.330370    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m97t\" (UniqueName: \"kubernetes.io/projected/f41cd0e8-cba6-4037-8e7d-2af3a82182ac-kube-api-access-9m97t\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 19:42:30.330519 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:30.330401    2562 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f41cd0e8-cba6-4037-8e7d-2af3a82182ac-config-file\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 19:42:31.153169 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:31.153131    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5wcfv"
Jul 06 19:42:31.153169 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:31.153153    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5wcfv" event={"ID":"f41cd0e8-cba6-4037-8e7d-2af3a82182ac","Type":"ContainerDied","Data":"551428f1c9badda6a0db86648511ff03f540b737e7366a2d2af91576bf668823"}
Jul 06 19:42:31.153761 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:31.153238    2562 scope.go:117] "RemoveContainer" containerID="b64956862ca6dec0c9aaff409ccce66d6cab1857d85da306664b7fd3a343d8cd"
Jul 06 19:42:31.186501 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:31.186462    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5wcfv"]
Jul 06 19:42:31.190046 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:31.190020    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5wcfv"]
Jul 06 19:42:31.893837 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:31.893809    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f41cd0e8-cba6-4037-8e7d-2af3a82182ac" path="/var/lib/kubelet/pods/f41cd0e8-cba6-4037-8e7d-2af3a82182ac/volumes"
Jul 06 19:42:35.700372 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:35.700339    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-52wzp"]
Jul 06 19:42:35.707207 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:35.700777    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f41cd0e8-cba6-4037-8e7d-2af3a82182ac" containerName="limitador"
Jul 06 19:42:35.707207 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:35.700795    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f41cd0e8-cba6-4037-8e7d-2af3a82182ac" containerName="limitador"
Jul 06 19:42:35.707207 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:35.700854    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f41cd0e8-cba6-4037-8e7d-2af3a82182ac" containerName="limitador"
Jul 06 19:42:35.707952 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:35.707930    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-52wzp"
Jul 06 19:42:35.716350 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:35.716331    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rfdpl\""
Jul 06 19:42:35.716474 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:35.716463    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\""
Jul 06 19:42:35.727533 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:35.727506    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-52wzp"]
Jul 06 19:42:35.762304 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:35.762281    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:35.769726 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:35.769708    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr5fn\" (UniqueName: \"kubernetes.io/projected/05e9136b-d59c-4d3a-8376-46cfe917a1a4-kube-api-access-gr5fn\") pod \"postgres-69fb4594cf-52wzp\" (UID: \"05e9136b-d59c-4d3a-8376-46cfe917a1a4\") " pod="opendatahub/postgres-69fb4594cf-52wzp"
Jul 06 19:42:35.769805 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:35.769759    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/05e9136b-d59c-4d3a-8376-46cfe917a1a4-data\") pod \"postgres-69fb4594cf-52wzp\" (UID: \"05e9136b-d59c-4d3a-8376-46cfe917a1a4\") " pod="opendatahub/postgres-69fb4594cf-52wzp"
Jul 06 19:42:35.870603 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:35.870578    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr5fn\" (UniqueName: \"kubernetes.io/projected/05e9136b-d59c-4d3a-8376-46cfe917a1a4-kube-api-access-gr5fn\") pod \"postgres-69fb4594cf-52wzp\" (UID: \"05e9136b-d59c-4d3a-8376-46cfe917a1a4\") " pod="opendatahub/postgres-69fb4594cf-52wzp"
Jul 06 19:42:35.870689 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:35.870616    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/05e9136b-d59c-4d3a-8376-46cfe917a1a4-data\") pod \"postgres-69fb4594cf-52wzp\" (UID: \"05e9136b-d59c-4d3a-8376-46cfe917a1a4\") " pod="opendatahub/postgres-69fb4594cf-52wzp"
Jul 06 19:42:35.870908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:35.870894    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/05e9136b-d59c-4d3a-8376-46cfe917a1a4-data\") pod \"postgres-69fb4594cf-52wzp\" (UID: \"05e9136b-d59c-4d3a-8376-46cfe917a1a4\") " pod="opendatahub/postgres-69fb4594cf-52wzp"
Jul 06 19:42:35.882462 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:35.882439    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr5fn\" (UniqueName: \"kubernetes.io/projected/05e9136b-d59c-4d3a-8376-46cfe917a1a4-kube-api-access-gr5fn\") pod \"postgres-69fb4594cf-52wzp\" (UID: \"05e9136b-d59c-4d3a-8376-46cfe917a1a4\") " pod="opendatahub/postgres-69fb4594cf-52wzp"
Jul 06 19:42:36.018016 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:36.017997    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-52wzp"
Jul 06 19:42:36.166379 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:36.166272    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-52wzp"]
Jul 06 19:42:36.168988 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:42:36.168949    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e9136b_d59c_4d3a_8376_46cfe917a1a4.slice/crio-269378b038e08b8ff4b67b9d8abfc40c65f5ae6f12291a94176cd0a4873e8c6d WatchSource:0}: Error finding container 269378b038e08b8ff4b67b9d8abfc40c65f5ae6f12291a94176cd0a4873e8c6d: Status 404 returned error can't find the container with id 269378b038e08b8ff4b67b9d8abfc40c65f5ae6f12291a94176cd0a4873e8c6d
Jul 06 19:42:37.180532 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:37.180475    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-52wzp" event={"ID":"05e9136b-d59c-4d3a-8376-46cfe917a1a4","Type":"ContainerStarted","Data":"269378b038e08b8ff4b67b9d8abfc40c65f5ae6f12291a94176cd0a4873e8c6d"}
Jul 06 19:42:42.203462 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:42.203429    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-52wzp" event={"ID":"05e9136b-d59c-4d3a-8376-46cfe917a1a4","Type":"ContainerStarted","Data":"15ba50adf5558f3d6cdab9165d742a74203f04460fa63215d4575bd5ce422bbd"}
Jul 06 19:42:42.203937 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:42.203565    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-52wzp"
Jul 06 19:42:42.224279 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:42.224234    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-52wzp" podStartSLOduration=2.178715916 podStartE2EDuration="7.224218868s" totalImagesPullingTime="5.045502952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:42:42.222345733 +0000 UTC m=+610.763522409" watchObservedRunningTime="2026-07-06 19:42:42.224218868 +0000 UTC m=+610.765395543"
Jul 06 19:42:48.238635 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:48.238603    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-52wzp"
Jul 06 19:42:48.618874 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:48.618841    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:48.952024 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:48.951946    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:55.939055 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:55.938967    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:56.022331 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.022262    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:56.167306 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.167277    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:56.233088 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.233024    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b7688899b-b9s2t"]
Jul 06 19:42:56.236152 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.236133    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b7688899b-b9s2t"
Jul 06 19:42:56.244563 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.244538    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 06 19:42:56.248351 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.248333    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 06 19:42:56.249697 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.249683    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rwct9\""
Jul 06 19:42:56.264858 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.264827    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b7688899b-b9s2t"]
Jul 06 19:42:56.313398 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.313377    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd838692-b497-4aac-8768-b580229242bf-webhook-cert\") pod \"maas-controller-6b7688899b-b9s2t\" (UID: \"dd838692-b497-4aac-8768-b580229242bf\") " pod="opendatahub/maas-controller-6b7688899b-b9s2t"
Jul 06 19:42:56.313525 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.313408    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rcrq\" (UniqueName: \"kubernetes.io/projected/dd838692-b497-4aac-8768-b580229242bf-kube-api-access-7rcrq\") pod \"maas-controller-6b7688899b-b9s2t\" (UID: \"dd838692-b497-4aac-8768-b580229242bf\") " pod="opendatahub/maas-controller-6b7688899b-b9s2t"
Jul 06 19:42:56.395194 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.395166    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:56.414198 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.414175    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd838692-b497-4aac-8768-b580229242bf-webhook-cert\") pod \"maas-controller-6b7688899b-b9s2t\" (UID: \"dd838692-b497-4aac-8768-b580229242bf\") " pod="opendatahub/maas-controller-6b7688899b-b9s2t"
Jul 06 19:42:56.414285 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.414207    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rcrq\" (UniqueName: \"kubernetes.io/projected/dd838692-b497-4aac-8768-b580229242bf-kube-api-access-7rcrq\") pod \"maas-controller-6b7688899b-b9s2t\" (UID: \"dd838692-b497-4aac-8768-b580229242bf\") " pod="opendatahub/maas-controller-6b7688899b-b9s2t"
Jul 06 19:42:56.414330 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:42:56.414315    2562 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 06 19:42:56.414384 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:42:56.414372    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd838692-b497-4aac-8768-b580229242bf-webhook-cert podName:dd838692-b497-4aac-8768-b580229242bf nodeName:}" failed. No retries permitted until 2026-07-06 19:42:56.914356009 +0000 UTC m=+625.455532663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/dd838692-b497-4aac-8768-b580229242bf-webhook-cert") pod "maas-controller-6b7688899b-b9s2t" (UID: "dd838692-b497-4aac-8768-b580229242bf") : secret "maas-controller-webhook-cert" not found
Jul 06 19:42:56.440872 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.440844    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rcrq\" (UniqueName: \"kubernetes.io/projected/dd838692-b497-4aac-8768-b580229242bf-kube-api-access-7rcrq\") pod \"maas-controller-6b7688899b-b9s2t\" (UID: \"dd838692-b497-4aac-8768-b580229242bf\") " pod="opendatahub/maas-controller-6b7688899b-b9s2t"
Jul 06 19:42:56.559169 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.559142    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-756d996fff-4kpwr"]
Jul 06 19:42:56.562303 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.562290    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-756d996fff-4kpwr"
Jul 06 19:42:56.577289 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.577262    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:56.588391 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.588366    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-756d996fff-4kpwr"]
Jul 06 19:42:56.703091 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.703060    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:42:56.716143 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.716120    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zqck\" (UniqueName: \"kubernetes.io/projected/5852b1ef-a193-4f80-8ea1-f73009c932ee-kube-api-access-7zqck\") pod \"maas-controller-756d996fff-4kpwr\" (UID: \"5852b1ef-a193-4f80-8ea1-f73009c932ee\") " pod="opendatahub/maas-controller-756d996fff-4kpwr"
Jul 06 19:42:56.716226 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.716173    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5852b1ef-a193-4f80-8ea1-f73009c932ee-webhook-cert\") pod \"maas-controller-756d996fff-4kpwr\" (UID: \"5852b1ef-a193-4f80-8ea1-f73009c932ee\") " pod="opendatahub/maas-controller-756d996fff-4kpwr"
Jul 06 19:42:56.816910 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.816843    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zqck\" (UniqueName: \"kubernetes.io/projected/5852b1ef-a193-4f80-8ea1-f73009c932ee-kube-api-access-7zqck\") pod \"maas-controller-756d996fff-4kpwr\" (UID: \"5852b1ef-a193-4f80-8ea1-f73009c932ee\") " pod="opendatahub/maas-controller-756d996fff-4kpwr"
Jul 06 19:42:56.817015 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.816915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5852b1ef-a193-4f80-8ea1-f73009c932ee-webhook-cert\") pod \"maas-controller-756d996fff-4kpwr\" (UID: \"5852b1ef-a193-4f80-8ea1-f73009c932ee\") " pod="opendatahub/maas-controller-756d996fff-4kpwr"
Jul 06 19:42:56.819314 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.819292    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5852b1ef-a193-4f80-8ea1-f73009c932ee-webhook-cert\") pod \"maas-controller-756d996fff-4kpwr\" (UID: \"5852b1ef-a193-4f80-8ea1-f73009c932ee\") " pod="opendatahub/maas-controller-756d996fff-4kpwr"
Jul 06 19:42:56.829883 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.829859    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zqck\" (UniqueName: \"kubernetes.io/projected/5852b1ef-a193-4f80-8ea1-f73009c932ee-kube-api-access-7zqck\") pod \"maas-controller-756d996fff-4kpwr\" (UID: \"5852b1ef-a193-4f80-8ea1-f73009c932ee\") " pod="opendatahub/maas-controller-756d996fff-4kpwr"
Jul 06 19:42:56.871393 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.871373    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-756d996fff-4kpwr"
Jul 06 19:42:56.917611 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.917583    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd838692-b497-4aac-8768-b580229242bf-webhook-cert\") pod \"maas-controller-6b7688899b-b9s2t\" (UID: \"dd838692-b497-4aac-8768-b580229242bf\") " pod="opendatahub/maas-controller-6b7688899b-b9s2t"
Jul 06 19:42:56.920705 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:56.920343    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd838692-b497-4aac-8768-b580229242bf-webhook-cert\") pod \"maas-controller-6b7688899b-b9s2t\" (UID: \"dd838692-b497-4aac-8768-b580229242bf\") " pod="opendatahub/maas-controller-6b7688899b-b9s2t"
Jul 06 19:42:57.010310 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:57.010287    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-756d996fff-4kpwr"]
Jul 06 19:42:57.012577 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:42:57.012548    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5852b1ef_a193_4f80_8ea1_f73009c932ee.slice/crio-7d17247e29c88b4cbd27eb92bb5818285be3c1b73a995ab0acf09bbb18e1bfff WatchSource:0}: Error finding container 7d17247e29c88b4cbd27eb92bb5818285be3c1b73a995ab0acf09bbb18e1bfff: Status 404 returned error can't find the container with id 7d17247e29c88b4cbd27eb92bb5818285be3c1b73a995ab0acf09bbb18e1bfff
Jul 06 19:42:57.145395 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:57.145330    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b7688899b-b9s2t"
Jul 06 19:42:57.267381 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:57.267352    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-756d996fff-4kpwr" event={"ID":"5852b1ef-a193-4f80-8ea1-f73009c932ee","Type":"ContainerStarted","Data":"7d17247e29c88b4cbd27eb92bb5818285be3c1b73a995ab0acf09bbb18e1bfff"}
Jul 06 19:42:57.282417 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:57.282396    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b7688899b-b9s2t"]
Jul 06 19:42:57.284884 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:42:57.284853    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd838692_b497_4aac_8768_b580229242bf.slice/crio-fb5c47895ee48e0b1bfd037764da4911f9d198c7f8f4aaf4d8beaf3bc8dd8120 WatchSource:0}: Error finding container fb5c47895ee48e0b1bfd037764da4911f9d198c7f8f4aaf4d8beaf3bc8dd8120: Status 404 returned error can't find the container with id fb5c47895ee48e0b1bfd037764da4911f9d198c7f8f4aaf4d8beaf3bc8dd8120
Jul 06 19:42:58.275760 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:42:58.275682    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b7688899b-b9s2t" event={"ID":"dd838692-b497-4aac-8768-b580229242bf","Type":"ContainerStarted","Data":"fb5c47895ee48e0b1bfd037764da4911f9d198c7f8f4aaf4d8beaf3bc8dd8120"}
Jul 06 19:43:00.148549 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:00.148461    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:43:00.285873 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:00.285838    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:43:00.287012 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:00.286985    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-756d996fff-4kpwr" event={"ID":"5852b1ef-a193-4f80-8ea1-f73009c932ee","Type":"ContainerStarted","Data":"04b405555cf0b6c87e591d89e60bc197d7d32a012c02e9ff3619afdca1c202e4"}
Jul 06 19:43:00.287155 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:00.287045    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-756d996fff-4kpwr"
Jul 06 19:43:00.288379 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:00.288355    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b7688899b-b9s2t" event={"ID":"dd838692-b497-4aac-8768-b580229242bf","Type":"ContainerStarted","Data":"5500dfd131652d9310c4b710628e8b69d28c76f654a9968fb1b1d0f780f39293"}
Jul 06 19:43:00.288467 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:00.288403    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b7688899b-b9s2t"
Jul 06 19:43:00.313969 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:00.313916    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-756d996fff-4kpwr" podStartSLOduration=1.442737406 podStartE2EDuration="4.313896319s" totalImagesPullingTime="2.871158913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:43:00.313132015 +0000 UTC m=+628.854308691" watchObservedRunningTime="2026-07-06 19:43:00.313896319 +0000 UTC m=+628.855072996"
Jul 06 19:43:00.338643 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:00.338603    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6b7688899b-b9s2t" podStartSLOduration=1.73267828 podStartE2EDuration="4.338591024s" totalImagesPullingTime="2.605912744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:43:00.337419826 +0000 UTC m=+628.878596499" watchObservedRunningTime="2026-07-06 19:43:00.338591024 +0000 UTC m=+628.879767700"
Jul 06 19:43:11.301629 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:11.301593    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b7688899b-b9s2t"
Jul 06 19:43:11.302059 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:11.301666    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-756d996fff-4kpwr"
Jul 06 19:43:11.410908 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:11.410879    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b7688899b-b9s2t"]
Jul 06 19:43:11.411133 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:11.411100    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-6b7688899b-b9s2t" podUID="dd838692-b497-4aac-8768-b580229242bf" containerName="manager" containerID="cri-o://5500dfd131652d9310c4b710628e8b69d28c76f654a9968fb1b1d0f780f39293" gracePeriod=30
Jul 06 19:43:11.659845 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:11.659819    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b7688899b-b9s2t"
Jul 06 19:43:11.837673 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:11.837603    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd838692-b497-4aac-8768-b580229242bf-webhook-cert\") pod \"dd838692-b497-4aac-8768-b580229242bf\" (UID: \"dd838692-b497-4aac-8768-b580229242bf\") "
Jul 06 19:43:11.837673 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:11.837636    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rcrq\" (UniqueName: \"kubernetes.io/projected/dd838692-b497-4aac-8768-b580229242bf-kube-api-access-7rcrq\") pod \"dd838692-b497-4aac-8768-b580229242bf\" (UID: \"dd838692-b497-4aac-8768-b580229242bf\") "
Jul 06 19:43:11.839767 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:11.839738    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd838692-b497-4aac-8768-b580229242bf-kube-api-access-7rcrq" (OuterVolumeSpecName: "kube-api-access-7rcrq") pod "dd838692-b497-4aac-8768-b580229242bf" (UID: "dd838692-b497-4aac-8768-b580229242bf"). InnerVolumeSpecName "kube-api-access-7rcrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:43:11.839874 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:11.839763    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd838692-b497-4aac-8768-b580229242bf-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "dd838692-b497-4aac-8768-b580229242bf" (UID: "dd838692-b497-4aac-8768-b580229242bf"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:43:11.939092 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:11.939069    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd838692-b497-4aac-8768-b580229242bf-webhook-cert\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 19:43:11.939092 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:11.939090    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rcrq\" (UniqueName: \"kubernetes.io/projected/dd838692-b497-4aac-8768-b580229242bf-kube-api-access-7rcrq\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 19:43:12.341806 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:12.341772    2562 generic.go:356] "Generic (PLEG): container finished" podID="dd838692-b497-4aac-8768-b580229242bf" containerID="5500dfd131652d9310c4b710628e8b69d28c76f654a9968fb1b1d0f780f39293" exitCode=0
Jul 06 19:43:12.342240 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:12.341816    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b7688899b-b9s2t"
Jul 06 19:43:12.342240 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:12.341846    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b7688899b-b9s2t" event={"ID":"dd838692-b497-4aac-8768-b580229242bf","Type":"ContainerDied","Data":"5500dfd131652d9310c4b710628e8b69d28c76f654a9968fb1b1d0f780f39293"}
Jul 06 19:43:12.342240 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:12.341902    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b7688899b-b9s2t" event={"ID":"dd838692-b497-4aac-8768-b580229242bf","Type":"ContainerDied","Data":"fb5c47895ee48e0b1bfd037764da4911f9d198c7f8f4aaf4d8beaf3bc8dd8120"}
Jul 06 19:43:12.342240 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:12.341937    2562 scope.go:117] "RemoveContainer" containerID="5500dfd131652d9310c4b710628e8b69d28c76f654a9968fb1b1d0f780f39293"
Jul 06 19:43:12.350296 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:12.350267    2562 scope.go:117] "RemoveContainer" containerID="5500dfd131652d9310c4b710628e8b69d28c76f654a9968fb1b1d0f780f39293"
Jul 06 19:43:12.350570 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:43:12.350550    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5500dfd131652d9310c4b710628e8b69d28c76f654a9968fb1b1d0f780f39293\": container with ID starting with 5500dfd131652d9310c4b710628e8b69d28c76f654a9968fb1b1d0f780f39293 not found: ID does not exist" containerID="5500dfd131652d9310c4b710628e8b69d28c76f654a9968fb1b1d0f780f39293"
Jul 06 19:43:12.350639 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:12.350579    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5500dfd131652d9310c4b710628e8b69d28c76f654a9968fb1b1d0f780f39293"} err="failed to get container status \"5500dfd131652d9310c4b710628e8b69d28c76f654a9968fb1b1d0f780f39293\": rpc error: code = NotFound desc = could not find container \"5500dfd131652d9310c4b710628e8b69d28c76f654a9968fb1b1d0f780f39293\": container with ID starting with 5500dfd131652d9310c4b710628e8b69d28c76f654a9968fb1b1d0f780f39293 not found: ID does not exist"
Jul 06 19:43:12.377051 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:12.377027    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b7688899b-b9s2t"]
Jul 06 19:43:12.385617 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:12.385599    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b7688899b-b9s2t"]
Jul 06 19:43:13.894327 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:13.894288    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd838692-b497-4aac-8768-b580229242bf" path="/var/lib/kubelet/pods/dd838692-b497-4aac-8768-b580229242bf/volumes"
Jul 06 19:43:30.884703 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:30.884670    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:43:31.459380 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.459343    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:43:31.525631 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.525605    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-6968b6764c-967mv"]
Jul 06 19:43:31.525892 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.525881    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd838692-b497-4aac-8768-b580229242bf" containerName="manager"
Jul 06 19:43:31.525941 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.525893    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd838692-b497-4aac-8768-b580229242bf" containerName="manager"
Jul 06 19:43:31.525981 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.525960    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd838692-b497-4aac-8768-b580229242bf" containerName="manager"
Jul 06 19:43:31.532187 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.532168    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-6968b6764c-967mv"
Jul 06 19:43:31.535628 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.535608    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\""
Jul 06 19:43:31.553533 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.553510    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-6968b6764c-967mv"]
Jul 06 19:43:31.572246 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.572222    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da904a7-e89a-450f-9f7d-8c35722909e7-maas-api-tls\") pod \"maas-api-6968b6764c-967mv\" (UID: \"1da904a7-e89a-450f-9f7d-8c35722909e7\") " pod="opendatahub/maas-api-6968b6764c-967mv"
Jul 06 19:43:31.572362 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.572258    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv6cd\" (UniqueName: \"kubernetes.io/projected/1da904a7-e89a-450f-9f7d-8c35722909e7-kube-api-access-fv6cd\") pod \"maas-api-6968b6764c-967mv\" (UID: \"1da904a7-e89a-450f-9f7d-8c35722909e7\") " pod="opendatahub/maas-api-6968b6764c-967mv"
Jul 06 19:43:31.572425 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.572380    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da904a7-e89a-450f-9f7d-8c35722909e7-openshift-service-ca\") pod \"maas-api-6968b6764c-967mv\" (UID: \"1da904a7-e89a-450f-9f7d-8c35722909e7\") " pod="opendatahub/maas-api-6968b6764c-967mv"
Jul 06 19:43:31.582946 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.582910    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:43:31.673325 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.673301    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da904a7-e89a-450f-9f7d-8c35722909e7-openshift-service-ca\") pod \"maas-api-6968b6764c-967mv\" (UID: \"1da904a7-e89a-450f-9f7d-8c35722909e7\") " pod="opendatahub/maas-api-6968b6764c-967mv"
Jul 06 19:43:31.673429 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.673348    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da904a7-e89a-450f-9f7d-8c35722909e7-maas-api-tls\") pod \"maas-api-6968b6764c-967mv\" (UID: \"1da904a7-e89a-450f-9f7d-8c35722909e7\") " pod="opendatahub/maas-api-6968b6764c-967mv"
Jul 06 19:43:31.673429 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.673365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv6cd\" (UniqueName: \"kubernetes.io/projected/1da904a7-e89a-450f-9f7d-8c35722909e7-kube-api-access-fv6cd\") pod \"maas-api-6968b6764c-967mv\" (UID: \"1da904a7-e89a-450f-9f7d-8c35722909e7\") " pod="opendatahub/maas-api-6968b6764c-967mv"
Jul 06 19:43:31.674242 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.674221    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da904a7-e89a-450f-9f7d-8c35722909e7-openshift-service-ca\") pod \"maas-api-6968b6764c-967mv\" (UID: \"1da904a7-e89a-450f-9f7d-8c35722909e7\") " pod="opendatahub/maas-api-6968b6764c-967mv"
Jul 06 19:43:31.675920 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.675897    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da904a7-e89a-450f-9f7d-8c35722909e7-maas-api-tls\") pod \"maas-api-6968b6764c-967mv\" (UID: \"1da904a7-e89a-450f-9f7d-8c35722909e7\") " pod="opendatahub/maas-api-6968b6764c-967mv"
Jul 06 19:43:31.689934 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.689914    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv6cd\" (UniqueName: \"kubernetes.io/projected/1da904a7-e89a-450f-9f7d-8c35722909e7-kube-api-access-fv6cd\") pod \"maas-api-6968b6764c-967mv\" (UID: \"1da904a7-e89a-450f-9f7d-8c35722909e7\") " pod="opendatahub/maas-api-6968b6764c-967mv"
Jul 06 19:43:31.842122 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.842083    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-6968b6764c-967mv"
Jul 06 19:43:31.996779 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:31.996659    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-6968b6764c-967mv"]
Jul 06 19:43:31.999915 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:43:31.999885    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da904a7_e89a_450f_9f7d_8c35722909e7.slice/crio-2c00654043e9dc1d6c28139b414642024cfeddb5d81e3ac661f1f9b7d50d2bad WatchSource:0}: Error finding container 2c00654043e9dc1d6c28139b414642024cfeddb5d81e3ac661f1f9b7d50d2bad: Status 404 returned error can't find the container with id 2c00654043e9dc1d6c28139b414642024cfeddb5d81e3ac661f1f9b7d50d2bad
Jul 06 19:43:32.356910 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:32.356879    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:43:32.430301 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:32.430270    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-6968b6764c-967mv" event={"ID":"1da904a7-e89a-450f-9f7d-8c35722909e7","Type":"ContainerStarted","Data":"2c00654043e9dc1d6c28139b414642024cfeddb5d81e3ac661f1f9b7d50d2bad"}
Jul 06 19:43:32.495579 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:32.495543    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:43:32.621897 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:32.621833    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-s8v5b"]
Jul 06 19:43:32.626423 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:32.626406    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-s8v5b"
Jul 06 19:43:32.629288 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:32.629270    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ztsvn\""
Jul 06 19:43:32.629778 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:32.629760    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 06 19:43:32.652796 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:32.652768    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-s8v5b"]
Jul 06 19:43:32.681976 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:32.681908    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlnlq\" (UniqueName: \"kubernetes.io/projected/248c632a-ba2a-40bf-a72e-0610ee5f87bd-kube-api-access-mlnlq\") pod \"payload-pre-processing-7964457696-s8v5b\" (UID: \"248c632a-ba2a-40bf-a72e-0610ee5f87bd\") " pod="openshift-ingress/payload-pre-processing-7964457696-s8v5b"
Jul 06 19:43:32.682073 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:32.681979    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248c632a-ba2a-40bf-a72e-0610ee5f87bd-plugins-config-volume\") pod \"payload-pre-processing-7964457696-s8v5b\" (UID: \"248c632a-ba2a-40bf-a72e-0610ee5f87bd\") " pod="openshift-ingress/payload-pre-processing-7964457696-s8v5b"
Jul 06 19:43:32.782594 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:32.782561    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248c632a-ba2a-40bf-a72e-0610ee5f87bd-plugins-config-volume\") pod \"payload-pre-processing-7964457696-s8v5b\" (UID: \"248c632a-ba2a-40bf-a72e-0610ee5f87bd\") " pod="openshift-ingress/payload-pre-processing-7964457696-s8v5b"
Jul 06 19:43:32.782765 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:32.782633    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlnlq\" (UniqueName: \"kubernetes.io/projected/248c632a-ba2a-40bf-a72e-0610ee5f87bd-kube-api-access-mlnlq\") pod \"payload-pre-processing-7964457696-s8v5b\" (UID: \"248c632a-ba2a-40bf-a72e-0610ee5f87bd\") " pod="openshift-ingress/payload-pre-processing-7964457696-s8v5b"
Jul 06 19:43:32.783519 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:32.783469    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/248c632a-ba2a-40bf-a72e-0610ee5f87bd-plugins-config-volume\") pod \"payload-pre-processing-7964457696-s8v5b\" (UID: \"248c632a-ba2a-40bf-a72e-0610ee5f87bd\") " pod="openshift-ingress/payload-pre-processing-7964457696-s8v5b"
Jul 06 19:43:32.827451 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:32.827424    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlnlq\" (UniqueName: \"kubernetes.io/projected/248c632a-ba2a-40bf-a72e-0610ee5f87bd-kube-api-access-mlnlq\") pod \"payload-pre-processing-7964457696-s8v5b\" (UID: \"248c632a-ba2a-40bf-a72e-0610ee5f87bd\") " pod="openshift-ingress/payload-pre-processing-7964457696-s8v5b"
Jul 06 19:43:32.936980 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:32.936919    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-s8v5b"
Jul 06 19:43:33.138674 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:33.138652    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-s8v5b"]
Jul 06 19:43:33.141009 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:43:33.140980    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248c632a_ba2a_40bf_a72e_0610ee5f87bd.slice/crio-dc38af21567806cf483b47546b94498a6268eb367bace2ddbb13c603cee26a97 WatchSource:0}: Error finding container dc38af21567806cf483b47546b94498a6268eb367bace2ddbb13c603cee26a97: Status 404 returned error can't find the container with id dc38af21567806cf483b47546b94498a6268eb367bace2ddbb13c603cee26a97
Jul 06 19:43:33.437336 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:33.437300    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-s8v5b" event={"ID":"248c632a-ba2a-40bf-a72e-0610ee5f87bd","Type":"ContainerStarted","Data":"dc38af21567806cf483b47546b94498a6268eb367bace2ddbb13c603cee26a97"}
Jul 06 19:43:34.443079 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:34.443043    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-6968b6764c-967mv" event={"ID":"1da904a7-e89a-450f-9f7d-8c35722909e7","Type":"ContainerStarted","Data":"c5406e019ffccbeb136a458d0809da65227cb7a7a2ee923e4d849f0c66111f22"}
Jul 06 19:43:34.443513 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:34.443161    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-6968b6764c-967mv"
Jul 06 19:43:34.478113 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:34.478069    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-6968b6764c-967mv" podStartSLOduration=1.723772287 podStartE2EDuration="3.478042298s" totalImagesPullingTime="1.754270011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:43:34.47783668 +0000 UTC m=+663.019013358" watchObservedRunningTime="2026-07-06 19:43:34.478042298 +0000 UTC m=+663.019218973"
Jul 06 19:43:35.448358 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:35.448312    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-s8v5b" event={"ID":"248c632a-ba2a-40bf-a72e-0610ee5f87bd","Type":"ContainerStarted","Data":"bef0948786f07d7ec23c4c546a0808089a545870be4d849322e12d2a401fb1e5"}
Jul 06 19:43:35.448749 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:35.448622    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-s8v5b"
Jul 06 19:43:35.473615 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:35.473572    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-s8v5b" podStartSLOduration=2.162347245 podStartE2EDuration="3.473559061s" totalImagesPullingTime="1.311211816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:43:35.469211094 +0000 UTC m=+664.010387772" watchObservedRunningTime="2026-07-06 19:43:35.473559061 +0000 UTC m=+664.014735736"
Jul 06 19:43:40.453789 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:40.453762    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-6968b6764c-967mv"
Jul 06 19:43:41.490817 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:41.490786    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:43:41.580458 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:41.580419    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:43:42.379019 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:42.378984    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:43:45.781616 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:45.781587    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:43:46.454732 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:46.454695    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7964457696-s8v5b" podUID="248c632a-ba2a-40bf-a72e-0610ee5f87bd" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.27:9004: connect: connection refused"
Jul 06 19:43:51.690763 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.690729    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"]
Jul 06 19:43:51.694326 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.694301    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:51.700638 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.700612    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rq94b\""
Jul 06 19:43:51.700760 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.700663    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 06 19:43:51.700760 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.700732    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 06 19:43:51.701623 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.701604    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 06 19:43:51.710847 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.710826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs\" (UID: \"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:51.710942 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.710866    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs\" (UID: \"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:51.710942 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.710892    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7zst\" (UniqueName: \"kubernetes.io/projected/eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122-kube-api-access-n7zst\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs\" (UID: \"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:51.711013 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.710948    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs\" (UID: \"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:51.711013 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.710963    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs\" (UID: \"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:51.711013 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.710987    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs\" (UID: \"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:51.713947 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.713926    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"]
Jul 06 19:43:51.745639 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.745612    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:43:51.812227 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.812205    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7zst\" (UniqueName: \"kubernetes.io/projected/eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122-kube-api-access-n7zst\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs\" (UID: \"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:51.812314 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.812235    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs\" (UID: \"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:51.812314 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.812254    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs\" (UID: \"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:51.812402 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.812314    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs\" (UID: \"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:51.812479 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.812461    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs\" (UID: \"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:51.812582 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.812554    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs\" (UID: \"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:51.812659 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.812630    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs\" (UID: \"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:51.812745 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.812667    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs\" (UID: \"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:51.812848 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.812827    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs\" (UID: \"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:51.814512 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.814466    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs\" (UID: \"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:51.814795 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.814773    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs\" (UID: \"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:51.824554 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:51.824525    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7zst\" (UniqueName: \"kubernetes.io/projected/eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122-kube-api-access-n7zst\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs\" (UID: \"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:52.004088 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:52.004063    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:43:52.141510 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:52.141473    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"]
Jul 06 19:43:52.143821 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:43:52.143799    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb5b2c35_ad8e_46a1_b5ba_bb9a400c4122.slice/crio-17254463165f826a36ac012b6583e71cbd8a35a365323adcee76120dafade53c WatchSource:0}: Error finding container 17254463165f826a36ac012b6583e71cbd8a35a365323adcee76120dafade53c: Status 404 returned error can't find the container with id 17254463165f826a36ac012b6583e71cbd8a35a365323adcee76120dafade53c
Jul 06 19:43:52.522451 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:52.522416    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs" event={"ID":"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122","Type":"ContainerStarted","Data":"17254463165f826a36ac012b6583e71cbd8a35a365323adcee76120dafade53c"}
Jul 06 19:43:53.528240 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:53.528203    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs" event={"ID":"eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122","Type":"ContainerStarted","Data":"4892e43cf935ec43daa278a841796206a7b04d4014aa60d70f388c4960d2df10"}
Jul 06 19:43:56.453930 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:56.453900    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-s8v5b"
Jul 06 19:43:56.486225 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:43:56.486178    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs" podStartSLOduration=4.47258702 podStartE2EDuration="5.486158961s" totalImagesPullingTime="1.013571941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:43:53.575129832 +0000 UTC m=+682.116306486" watchObservedRunningTime="2026-07-06 19:43:56.486158961 +0000 UTC m=+685.027335637"
Jul 06 19:44:02.004190 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:44:02.004148    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:44:02.004669 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:44:02.004202    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:44:02.017395 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:44:02.017375    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:44:02.575883 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:44:02.575859    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs"
Jul 06 19:44:11.435715 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:44:11.435679    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:44:25.621937 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:44:25.621864    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:44:31.296529 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:44:31.296473    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:45:00.179105 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:00.179072    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-lns4g"]
Jul 06 19:45:00.182234 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:00.182217    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-lns4g"
Jul 06 19:45:00.186233 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:00.186211    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-cmcxr\""
Jul 06 19:45:00.231771 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:00.231751    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-lns4g"]
Jul 06 19:45:00.292113 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:00.292085    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dzbz\" (UniqueName: \"kubernetes.io/projected/20b287a7-6de7-4a7d-b9d8-a286a8fe5e98-kube-api-access-8dzbz\") pod \"maas-api-key-cleanup-29722785-lns4g\" (UID: \"20b287a7-6de7-4a7d-b9d8-a286a8fe5e98\") " pod="opendatahub/maas-api-key-cleanup-29722785-lns4g"
Jul 06 19:45:00.392451 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:00.392422    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dzbz\" (UniqueName: \"kubernetes.io/projected/20b287a7-6de7-4a7d-b9d8-a286a8fe5e98-kube-api-access-8dzbz\") pod \"maas-api-key-cleanup-29722785-lns4g\" (UID: \"20b287a7-6de7-4a7d-b9d8-a286a8fe5e98\") " pod="opendatahub/maas-api-key-cleanup-29722785-lns4g"
Jul 06 19:45:00.405371 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:00.405349    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dzbz\" (UniqueName: \"kubernetes.io/projected/20b287a7-6de7-4a7d-b9d8-a286a8fe5e98-kube-api-access-8dzbz\") pod \"maas-api-key-cleanup-29722785-lns4g\" (UID: \"20b287a7-6de7-4a7d-b9d8-a286a8fe5e98\") " pod="opendatahub/maas-api-key-cleanup-29722785-lns4g"
Jul 06 19:45:00.492362 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:00.492302    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-lns4g"
Jul 06 19:45:00.830191 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:00.830161    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-lns4g"]
Jul 06 19:45:00.833162 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:45:00.833136    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b287a7_6de7_4a7d_b9d8_a286a8fe5e98.slice/crio-b62239e1f707372aab50118443e3f89aa943523280e5ef07bac654d51ff680be WatchSource:0}: Error finding container b62239e1f707372aab50118443e3f89aa943523280e5ef07bac654d51ff680be: Status 404 returned error can't find the container with id b62239e1f707372aab50118443e3f89aa943523280e5ef07bac654d51ff680be
Jul 06 19:45:00.835366 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:00.835350    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 19:45:01.812614 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:01.812582    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-lns4g" event={"ID":"20b287a7-6de7-4a7d-b9d8-a286a8fe5e98","Type":"ContainerStarted","Data":"b62239e1f707372aab50118443e3f89aa943523280e5ef07bac654d51ff680be"}
Jul 06 19:45:03.823236 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:03.823199    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-lns4g" event={"ID":"20b287a7-6de7-4a7d-b9d8-a286a8fe5e98","Type":"ContainerStarted","Data":"22d24f19216af5dfa6a6d002ee5a0dfb8342665933d4f4620079609b8a7b942e"}
Jul 06 19:45:03.843871 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:03.843830    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722785-lns4g" podStartSLOduration=1.4408090630000001 podStartE2EDuration="3.843815693s" totalImagesPullingTime="2.40300663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:45:03.842671373 +0000 UTC m=+752.383848049" watchObservedRunningTime="2026-07-06 19:45:03.843815693 +0000 UTC m=+752.384992370"
Jul 06 19:45:04.828734 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:04.828700    2562 generic.go:356] "Generic (PLEG): container finished" podID="20b287a7-6de7-4a7d-b9d8-a286a8fe5e98" containerID="22d24f19216af5dfa6a6d002ee5a0dfb8342665933d4f4620079609b8a7b942e" exitCode=0
Jul 06 19:45:04.829122 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:04.828789    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-lns4g" event={"ID":"20b287a7-6de7-4a7d-b9d8-a286a8fe5e98","Type":"ContainerDied","Data":"22d24f19216af5dfa6a6d002ee5a0dfb8342665933d4f4620079609b8a7b942e"}
Jul 06 19:45:05.962682 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:05.962659    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-lns4g"
Jul 06 19:45:06.134750 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:06.134685    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dzbz\" (UniqueName: \"kubernetes.io/projected/20b287a7-6de7-4a7d-b9d8-a286a8fe5e98-kube-api-access-8dzbz\") pod \"20b287a7-6de7-4a7d-b9d8-a286a8fe5e98\" (UID: \"20b287a7-6de7-4a7d-b9d8-a286a8fe5e98\") "
Jul 06 19:45:06.136606 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:06.136575    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b287a7-6de7-4a7d-b9d8-a286a8fe5e98-kube-api-access-8dzbz" (OuterVolumeSpecName: "kube-api-access-8dzbz") pod "20b287a7-6de7-4a7d-b9d8-a286a8fe5e98" (UID: "20b287a7-6de7-4a7d-b9d8-a286a8fe5e98"). InnerVolumeSpecName "kube-api-access-8dzbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:45:06.235388 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:06.235363    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dzbz\" (UniqueName: \"kubernetes.io/projected/20b287a7-6de7-4a7d-b9d8-a286a8fe5e98-kube-api-access-8dzbz\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 19:45:06.839674 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:06.839646    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-lns4g"
Jul 06 19:45:06.839674 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:06.839661    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-lns4g" event={"ID":"20b287a7-6de7-4a7d-b9d8-a286a8fe5e98","Type":"ContainerDied","Data":"b62239e1f707372aab50118443e3f89aa943523280e5ef07bac654d51ff680be"}
Jul 06 19:45:06.839861 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:06.839712    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b62239e1f707372aab50118443e3f89aa943523280e5ef07bac654d51ff680be"
Jul 06 19:45:09.009115 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.009087    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"]
Jul 06 19:45:09.009464 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.009398    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20b287a7-6de7-4a7d-b9d8-a286a8fe5e98" containerName="cleanup"
Jul 06 19:45:09.009464 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.009409    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="20b287a7-6de7-4a7d-b9d8-a286a8fe5e98" containerName="cleanup"
Jul 06 19:45:09.009464 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.009455    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="20b287a7-6de7-4a7d-b9d8-a286a8fe5e98" containerName="cleanup"
Jul 06 19:45:09.012384 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.012369    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.016465 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.016442    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 06 19:45:09.028348 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.028318    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"]
Jul 06 19:45:09.042861 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.042818    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:45:09.156999 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.156971    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e11e7b03-95ef-4d5a-896c-b433fc344f89-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7kxxk\" (UID: \"e11e7b03-95ef-4d5a-896c-b433fc344f89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.157133 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.157004    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsfg5\" (UniqueName: \"kubernetes.io/projected/e11e7b03-95ef-4d5a-896c-b433fc344f89-kube-api-access-nsfg5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7kxxk\" (UID: \"e11e7b03-95ef-4d5a-896c-b433fc344f89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.157133 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.157049    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e11e7b03-95ef-4d5a-896c-b433fc344f89-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7kxxk\" (UID: \"e11e7b03-95ef-4d5a-896c-b433fc344f89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.157133 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.157118    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e11e7b03-95ef-4d5a-896c-b433fc344f89-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7kxxk\" (UID: \"e11e7b03-95ef-4d5a-896c-b433fc344f89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.157293 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.157168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e11e7b03-95ef-4d5a-896c-b433fc344f89-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7kxxk\" (UID: \"e11e7b03-95ef-4d5a-896c-b433fc344f89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.157293 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.157220    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e11e7b03-95ef-4d5a-896c-b433fc344f89-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7kxxk\" (UID: \"e11e7b03-95ef-4d5a-896c-b433fc344f89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.257851 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.257826    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e11e7b03-95ef-4d5a-896c-b433fc344f89-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7kxxk\" (UID: \"e11e7b03-95ef-4d5a-896c-b433fc344f89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.257958 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.257864    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e11e7b03-95ef-4d5a-896c-b433fc344f89-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7kxxk\" (UID: \"e11e7b03-95ef-4d5a-896c-b433fc344f89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.257958 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.257885    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsfg5\" (UniqueName: \"kubernetes.io/projected/e11e7b03-95ef-4d5a-896c-b433fc344f89-kube-api-access-nsfg5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7kxxk\" (UID: \"e11e7b03-95ef-4d5a-896c-b433fc344f89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.258106 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.257995    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e11e7b03-95ef-4d5a-896c-b433fc344f89-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7kxxk\" (UID: \"e11e7b03-95ef-4d5a-896c-b433fc344f89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.258106 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.258029    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e11e7b03-95ef-4d5a-896c-b433fc344f89-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7kxxk\" (UID: \"e11e7b03-95ef-4d5a-896c-b433fc344f89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.258106 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.258056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e11e7b03-95ef-4d5a-896c-b433fc344f89-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7kxxk\" (UID: \"e11e7b03-95ef-4d5a-896c-b433fc344f89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.258383 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.258360    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e11e7b03-95ef-4d5a-896c-b433fc344f89-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7kxxk\" (UID: \"e11e7b03-95ef-4d5a-896c-b433fc344f89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.258476 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.258381    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e11e7b03-95ef-4d5a-896c-b433fc344f89-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7kxxk\" (UID: \"e11e7b03-95ef-4d5a-896c-b433fc344f89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.258476 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.258425    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e11e7b03-95ef-4d5a-896c-b433fc344f89-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7kxxk\" (UID: \"e11e7b03-95ef-4d5a-896c-b433fc344f89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.260121 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.260069    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e11e7b03-95ef-4d5a-896c-b433fc344f89-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7kxxk\" (UID: \"e11e7b03-95ef-4d5a-896c-b433fc344f89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.260323 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.260307    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e11e7b03-95ef-4d5a-896c-b433fc344f89-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7kxxk\" (UID: \"e11e7b03-95ef-4d5a-896c-b433fc344f89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.267889 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.267866    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsfg5\" (UniqueName: \"kubernetes.io/projected/e11e7b03-95ef-4d5a-896c-b433fc344f89-kube-api-access-nsfg5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7kxxk\" (UID: \"e11e7b03-95ef-4d5a-896c-b433fc344f89\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.322132 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.322107    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:09.455455 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.455426    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"]
Jul 06 19:45:09.458263 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:45:09.458235    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11e7b03_95ef_4d5a_896c_b433fc344f89.slice/crio-037c90ad7de48ec1afd097e172efb07d2dc30b65569d5e5853efac5a22eec1fe WatchSource:0}: Error finding container 037c90ad7de48ec1afd097e172efb07d2dc30b65569d5e5853efac5a22eec1fe: Status 404 returned error can't find the container with id 037c90ad7de48ec1afd097e172efb07d2dc30b65569d5e5853efac5a22eec1fe
Jul 06 19:45:09.854246 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.854218    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk" event={"ID":"e11e7b03-95ef-4d5a-896c-b433fc344f89","Type":"ContainerStarted","Data":"86819cdea48caee3257e5c104d5cc37abdbad15696ec41262075031bde729558"}
Jul 06 19:45:09.854378 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.854265    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk" event={"ID":"e11e7b03-95ef-4d5a-896c-b433fc344f89","Type":"ContainerStarted","Data":"037c90ad7de48ec1afd097e172efb07d2dc30b65569d5e5853efac5a22eec1fe"}
Jul 06 19:45:09.886026 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:09.885982    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk" podStartSLOduration=1.679854197 podStartE2EDuration="1.885969084s" totalImagesPullingTime="206.114887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:45:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:45:09.883036622 +0000 UTC m=+758.424213299" watchObservedRunningTime="2026-07-06 19:45:09.885969084 +0000 UTC m=+758.427145760"
Jul 06 19:45:11.226890 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:11.226810    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:45:19.322430 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:19.322394    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:19.322430 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:19.322441    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:19.334809 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:19.334785    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:19.913008 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:19.912980    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7kxxk"
Jul 06 19:45:20.401496 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.401448    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"]
Jul 06 19:45:20.405369 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.405349    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.413041 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.413017    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 06 19:45:20.441117 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.441095    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"]
Jul 06 19:45:20.467371 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.467342    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:45:20.539596 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.539572    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvjzw\" (UniqueName: \"kubernetes.io/projected/9cd5025e-fac4-414d-93e9-5add82837fe7-kube-api-access-pvjzw\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc\" (UID: \"9cd5025e-fac4-414d-93e9-5add82837fe7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.539704 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.539612    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cd5025e-fac4-414d-93e9-5add82837fe7-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc\" (UID: \"9cd5025e-fac4-414d-93e9-5add82837fe7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.539704 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.539642    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd5025e-fac4-414d-93e9-5add82837fe7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc\" (UID: \"9cd5025e-fac4-414d-93e9-5add82837fe7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.539704 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.539693    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd5025e-fac4-414d-93e9-5add82837fe7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc\" (UID: \"9cd5025e-fac4-414d-93e9-5add82837fe7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.539815 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.539719    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd5025e-fac4-414d-93e9-5add82837fe7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc\" (UID: \"9cd5025e-fac4-414d-93e9-5add82837fe7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.539815 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.539781    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd5025e-fac4-414d-93e9-5add82837fe7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc\" (UID: \"9cd5025e-fac4-414d-93e9-5add82837fe7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.640448 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.640425    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjzw\" (UniqueName: \"kubernetes.io/projected/9cd5025e-fac4-414d-93e9-5add82837fe7-kube-api-access-pvjzw\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc\" (UID: \"9cd5025e-fac4-414d-93e9-5add82837fe7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.640580 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.640462    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cd5025e-fac4-414d-93e9-5add82837fe7-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc\" (UID: \"9cd5025e-fac4-414d-93e9-5add82837fe7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.640647 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.640599    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd5025e-fac4-414d-93e9-5add82837fe7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc\" (UID: \"9cd5025e-fac4-414d-93e9-5add82837fe7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.640714 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.640660    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd5025e-fac4-414d-93e9-5add82837fe7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc\" (UID: \"9cd5025e-fac4-414d-93e9-5add82837fe7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.640714 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.640694    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd5025e-fac4-414d-93e9-5add82837fe7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc\" (UID: \"9cd5025e-fac4-414d-93e9-5add82837fe7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.640814 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.640723    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd5025e-fac4-414d-93e9-5add82837fe7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc\" (UID: \"9cd5025e-fac4-414d-93e9-5add82837fe7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.640857 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.640813    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cd5025e-fac4-414d-93e9-5add82837fe7-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc\" (UID: \"9cd5025e-fac4-414d-93e9-5add82837fe7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.640964 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.640945    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd5025e-fac4-414d-93e9-5add82837fe7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc\" (UID: \"9cd5025e-fac4-414d-93e9-5add82837fe7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.641126 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.641107    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd5025e-fac4-414d-93e9-5add82837fe7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc\" (UID: \"9cd5025e-fac4-414d-93e9-5add82837fe7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.642921 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.642893    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd5025e-fac4-414d-93e9-5add82837fe7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc\" (UID: \"9cd5025e-fac4-414d-93e9-5add82837fe7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.643224 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.643198    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd5025e-fac4-414d-93e9-5add82837fe7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc\" (UID: \"9cd5025e-fac4-414d-93e9-5add82837fe7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.652289 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.652237    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvjzw\" (UniqueName: \"kubernetes.io/projected/9cd5025e-fac4-414d-93e9-5add82837fe7-kube-api-access-pvjzw\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc\" (UID: \"9cd5025e-fac4-414d-93e9-5add82837fe7\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.714943 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.714916    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:20.846349 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.846317    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"]
Jul 06 19:45:20.846860 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:45:20.846826    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cd5025e_fac4_414d_93e9_5add82837fe7.slice/crio-7a5e892ed1012916e505513862b83f28ebd847440cc6716aa0e99dfe6a8e2d28 WatchSource:0}: Error finding container 7a5e892ed1012916e505513862b83f28ebd847440cc6716aa0e99dfe6a8e2d28: Status 404 returned error can't find the container with id 7a5e892ed1012916e505513862b83f28ebd847440cc6716aa0e99dfe6a8e2d28
Jul 06 19:45:20.911926 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:20.911859    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc" event={"ID":"9cd5025e-fac4-414d-93e9-5add82837fe7","Type":"ContainerStarted","Data":"7a5e892ed1012916e505513862b83f28ebd847440cc6716aa0e99dfe6a8e2d28"}
Jul 06 19:45:21.917975 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:21.917941    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc" event={"ID":"9cd5025e-fac4-414d-93e9-5add82837fe7","Type":"ContainerStarted","Data":"d377cdbb92d1e74ada965d8ad8d2170043c8871743031632ea0d20fbf539693d"}
Jul 06 19:45:21.949435 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:21.949387    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc" podStartSLOduration=1.743518637 podStartE2EDuration="1.949371602s" totalImagesPullingTime="205.852965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:45:21.948883925 +0000 UTC m=+770.490060601" watchObservedRunningTime="2026-07-06 19:45:21.949371602 +0000 UTC m=+770.490548277"
Jul 06 19:45:30.715430 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:30.715383    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:30.715913 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:30.715477    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:30.727772 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:30.727748    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:30.969787 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:30.969721    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc"
Jul 06 19:45:51.738678 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:51.738632    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:45:51.844180 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:45:51.844136    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:46:10.021437 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:46:10.021350    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:46:54.516839 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:46:54.516801    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:46:54.648669 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:46:54.648630    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:46:55.691406 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:46:55.691354    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:46:55.797247 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:46:55.797219    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:46:57.083985 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:46:57.083947    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:46:57.207539 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:46:57.207497    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:47:38.948965 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:47:38.948924    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:47:56.401117 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:47:56.401072    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:48:06.982061 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:06.982012    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:48:07.073792 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:07.073761    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-756d996fff-4kpwr"]
Jul 06 19:48:07.074052 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:07.074011    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-756d996fff-4kpwr" podUID="5852b1ef-a193-4f80-8ea1-f73009c932ee" containerName="manager" containerID="cri-o://04b405555cf0b6c87e591d89e60bc197d7d32a012c02e9ff3619afdca1c202e4" gracePeriod=30
Jul 06 19:48:07.320125 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:07.320099    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-756d996fff-4kpwr"
Jul 06 19:48:07.441638 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:07.441595    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5852b1ef-a193-4f80-8ea1-f73009c932ee-webhook-cert\") pod \"5852b1ef-a193-4f80-8ea1-f73009c932ee\" (UID: \"5852b1ef-a193-4f80-8ea1-f73009c932ee\") "
Jul 06 19:48:07.441638 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:07.441646    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zqck\" (UniqueName: \"kubernetes.io/projected/5852b1ef-a193-4f80-8ea1-f73009c932ee-kube-api-access-7zqck\") pod \"5852b1ef-a193-4f80-8ea1-f73009c932ee\" (UID: \"5852b1ef-a193-4f80-8ea1-f73009c932ee\") "
Jul 06 19:48:07.443530 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:07.443504    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5852b1ef-a193-4f80-8ea1-f73009c932ee-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5852b1ef-a193-4f80-8ea1-f73009c932ee" (UID: "5852b1ef-a193-4f80-8ea1-f73009c932ee"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:48:07.443530 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:07.443520    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5852b1ef-a193-4f80-8ea1-f73009c932ee-kube-api-access-7zqck" (OuterVolumeSpecName: "kube-api-access-7zqck") pod "5852b1ef-a193-4f80-8ea1-f73009c932ee" (UID: "5852b1ef-a193-4f80-8ea1-f73009c932ee"). InnerVolumeSpecName "kube-api-access-7zqck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:48:07.542831 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:07.542807    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5852b1ef-a193-4f80-8ea1-f73009c932ee-webhook-cert\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 19:48:07.542951 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:07.542833    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zqck\" (UniqueName: \"kubernetes.io/projected/5852b1ef-a193-4f80-8ea1-f73009c932ee-kube-api-access-7zqck\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 19:48:07.663182 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:07.663149    2562 generic.go:356] "Generic (PLEG): container finished" podID="5852b1ef-a193-4f80-8ea1-f73009c932ee" containerID="04b405555cf0b6c87e591d89e60bc197d7d32a012c02e9ff3619afdca1c202e4" exitCode=0
Jul 06 19:48:07.663307 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:07.663198    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-756d996fff-4kpwr"
Jul 06 19:48:07.663307 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:07.663248    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-756d996fff-4kpwr" event={"ID":"5852b1ef-a193-4f80-8ea1-f73009c932ee","Type":"ContainerDied","Data":"04b405555cf0b6c87e591d89e60bc197d7d32a012c02e9ff3619afdca1c202e4"}
Jul 06 19:48:07.663422 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:07.663313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-756d996fff-4kpwr" event={"ID":"5852b1ef-a193-4f80-8ea1-f73009c932ee","Type":"ContainerDied","Data":"7d17247e29c88b4cbd27eb92bb5818285be3c1b73a995ab0acf09bbb18e1bfff"}
Jul 06 19:48:07.663422 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:07.663347    2562 scope.go:117] "RemoveContainer" containerID="04b405555cf0b6c87e591d89e60bc197d7d32a012c02e9ff3619afdca1c202e4"
Jul 06 19:48:07.673082 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:07.673060    2562 scope.go:117] "RemoveContainer" containerID="04b405555cf0b6c87e591d89e60bc197d7d32a012c02e9ff3619afdca1c202e4"
Jul 06 19:48:07.673362 ip-10-0-128-200 kubenswrapper[2562]: E0706 19:48:07.673341    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04b405555cf0b6c87e591d89e60bc197d7d32a012c02e9ff3619afdca1c202e4\": container with ID starting with 04b405555cf0b6c87e591d89e60bc197d7d32a012c02e9ff3619afdca1c202e4 not found: ID does not exist" containerID="04b405555cf0b6c87e591d89e60bc197d7d32a012c02e9ff3619afdca1c202e4"
Jul 06 19:48:07.673411 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:07.673372    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b405555cf0b6c87e591d89e60bc197d7d32a012c02e9ff3619afdca1c202e4"} err="failed to get container status \"04b405555cf0b6c87e591d89e60bc197d7d32a012c02e9ff3619afdca1c202e4\": rpc error: code = NotFound desc = could not find container \"04b405555cf0b6c87e591d89e60bc197d7d32a012c02e9ff3619afdca1c202e4\": container with ID starting with 04b405555cf0b6c87e591d89e60bc197d7d32a012c02e9ff3619afdca1c202e4 not found: ID does not exist"
Jul 06 19:48:07.690354 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:07.690326    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-756d996fff-4kpwr"]
Jul 06 19:48:07.693160 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:07.693139    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-756d996fff-4kpwr"]
Jul 06 19:48:07.893941 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:07.893888    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5852b1ef-a193-4f80-8ea1-f73009c932ee" path="/var/lib/kubelet/pods/5852b1ef-a193-4f80-8ea1-f73009c932ee/volumes"
Jul 06 19:48:08.388920 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:08.388886    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:48:08.522764 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:08.522735    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-756d996fff-k5fsp"]
Jul 06 19:48:08.523039 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:08.523028    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5852b1ef-a193-4f80-8ea1-f73009c932ee" containerName="manager"
Jul 06 19:48:08.523080 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:08.523041    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5852b1ef-a193-4f80-8ea1-f73009c932ee" containerName="manager"
Jul 06 19:48:08.523123 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:08.523113    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5852b1ef-a193-4f80-8ea1-f73009c932ee" containerName="manager"
Jul 06 19:48:08.527317 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:08.527300    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-756d996fff-k5fsp"
Jul 06 19:48:08.535478 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:08.535448    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 06 19:48:08.535478 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:08.535474    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 06 19:48:08.535689 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:08.535474    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rwct9\""
Jul 06 19:48:08.553260 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:08.553232    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-756d996fff-k5fsp"]
Jul 06 19:48:08.649838 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:08.649784    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcdgq\" (UniqueName: \"kubernetes.io/projected/cb2e423c-8346-4f6d-9d59-90cc9b012ba7-kube-api-access-wcdgq\") pod \"maas-controller-756d996fff-k5fsp\" (UID: \"cb2e423c-8346-4f6d-9d59-90cc9b012ba7\") " pod="opendatahub/maas-controller-756d996fff-k5fsp"
Jul 06 19:48:08.649838 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:08.649829    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb2e423c-8346-4f6d-9d59-90cc9b012ba7-webhook-cert\") pod \"maas-controller-756d996fff-k5fsp\" (UID: \"cb2e423c-8346-4f6d-9d59-90cc9b012ba7\") " pod="opendatahub/maas-controller-756d996fff-k5fsp"
Jul 06 19:48:08.751206 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:08.751179    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcdgq\" (UniqueName: \"kubernetes.io/projected/cb2e423c-8346-4f6d-9d59-90cc9b012ba7-kube-api-access-wcdgq\") pod \"maas-controller-756d996fff-k5fsp\" (UID: \"cb2e423c-8346-4f6d-9d59-90cc9b012ba7\") " pod="opendatahub/maas-controller-756d996fff-k5fsp"
Jul 06 19:48:08.751297 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:08.751234    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb2e423c-8346-4f6d-9d59-90cc9b012ba7-webhook-cert\") pod \"maas-controller-756d996fff-k5fsp\" (UID: \"cb2e423c-8346-4f6d-9d59-90cc9b012ba7\") " pod="opendatahub/maas-controller-756d996fff-k5fsp"
Jul 06 19:48:08.753770 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:08.753742    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb2e423c-8346-4f6d-9d59-90cc9b012ba7-webhook-cert\") pod \"maas-controller-756d996fff-k5fsp\" (UID: \"cb2e423c-8346-4f6d-9d59-90cc9b012ba7\") " pod="opendatahub/maas-controller-756d996fff-k5fsp"
Jul 06 19:48:08.761808 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:08.761786    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcdgq\" (UniqueName: \"kubernetes.io/projected/cb2e423c-8346-4f6d-9d59-90cc9b012ba7-kube-api-access-wcdgq\") pod \"maas-controller-756d996fff-k5fsp\" (UID: \"cb2e423c-8346-4f6d-9d59-90cc9b012ba7\") " pod="opendatahub/maas-controller-756d996fff-k5fsp"
Jul 06 19:48:08.837836 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:08.837809    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-756d996fff-k5fsp"
Jul 06 19:48:08.976314 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:08.976292    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-756d996fff-k5fsp"]
Jul 06 19:48:08.977860 ip-10-0-128-200 kubenswrapper[2562]: W0706 19:48:08.977831    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2e423c_8346_4f6d_9d59_90cc9b012ba7.slice/crio-e0f4d3cc4d0582fc17027bf1f6766b7cacd44cdbf5bea1d98599ed0393664fcd WatchSource:0}: Error finding container e0f4d3cc4d0582fc17027bf1f6766b7cacd44cdbf5bea1d98599ed0393664fcd: Status 404 returned error can't find the container with id e0f4d3cc4d0582fc17027bf1f6766b7cacd44cdbf5bea1d98599ed0393664fcd
Jul 06 19:48:09.674277 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:09.674245    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-756d996fff-k5fsp" event={"ID":"cb2e423c-8346-4f6d-9d59-90cc9b012ba7","Type":"ContainerStarted","Data":"c7694a470afaf5c3f154237bdf3b7659e977f3debbdf0ee0ee31178e1ba55679"}
Jul 06 19:48:09.674627 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:09.674334    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-756d996fff-k5fsp" event={"ID":"cb2e423c-8346-4f6d-9d59-90cc9b012ba7","Type":"ContainerStarted","Data":"e0f4d3cc4d0582fc17027bf1f6766b7cacd44cdbf5bea1d98599ed0393664fcd"}
Jul 06 19:48:09.674627 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:09.674366    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-756d996fff-k5fsp"
Jul 06 19:48:09.709369 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:09.709324    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-756d996fff-k5fsp" podStartSLOduration=1.096739049 podStartE2EDuration="1.709311013s" totalImagesPullingTime="612.571964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:48:09.709304222 +0000 UTC m=+938.250480899" watchObservedRunningTime="2026-07-06 19:48:09.709311013 +0000 UTC m=+938.250487688"
Jul 06 19:48:20.685266 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:20.685233    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-756d996fff-k5fsp"
Jul 06 19:48:52.137367 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:52.137329    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:48:52.235023 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:52.234999    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:48:52.382533 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:48:52.382469    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:49:00.382355 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:49:00.382260    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:49:00.498349 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:49:00.498310    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:49:00.623359 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:49:00.623330    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:49:00.713568 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:49:00.713474    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:49:00.846655 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:49:00.846628    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:49:07.529841 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:49:07.529799    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:49:07.633083 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:49:07.633044    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:52:11.352620 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:52:11.352530    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:53:23.461928 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:53:23.461883    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:53:39.575641 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:53:39.575598    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:56:39.699770 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:56:39.699735    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:56:39.778232 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:56:39.778203    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:56:39.905731 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:56:39.905696    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:56:40.391814 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:56:40.391778    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:56:40.487981 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:56:40.487932    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:56:50.348162 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:56:50.348128    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:56:50.510998 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:56:50.510961    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:56:50.642314 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:56:50.642249    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:57:55.897462 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:57:55.897313    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:57:57.248601 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:57:57.248545    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 19:59:56.265112 ip-10-0-128-200 kubenswrapper[2562]: I0706 19:59:56.265078    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:03.684589 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:03.684548    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:11.483085 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:11.483043    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:16.536924 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:16.536891    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:19.521884 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:19.521844    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:19.629969 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:19.629940    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:19.992663 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:19.992573    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:26.613705 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:26.613656    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:26.718312 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:26.718273    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:26.872396 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:26.872317    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:26.991237 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:26.991208    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:28.060916 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.060882    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:28.165056 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.165027    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:28.267010 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.266982    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"]
Jul 06 20:00:28.275857 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.275835    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"
Jul 06 20:00:28.281771 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.281749    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-cmcxr\""
Jul 06 20:00:28.282079 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.282053    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-53be5925-serving-cert\""
Jul 06 20:00:28.287526 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.287502    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"]
Jul 06 20:00:28.329341 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.329287    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/761d62ea-7297-4b7c-8009-28f4782541a9-maas-api-tls\") pod \"maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8\" (UID: \"761d62ea-7297-4b7c-8009-28f4782541a9\") " pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"
Jul 06 20:00:28.329341 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.329321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/761d62ea-7297-4b7c-8009-28f4782541a9-openshift-service-ca\") pod \"maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8\" (UID: \"761d62ea-7297-4b7c-8009-28f4782541a9\") " pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"
Jul 06 20:00:28.329532 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.329361    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsq52\" (UniqueName: \"kubernetes.io/projected/761d62ea-7297-4b7c-8009-28f4782541a9-kube-api-access-tsq52\") pod \"maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8\" (UID: \"761d62ea-7297-4b7c-8009-28f4782541a9\") " pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"
Jul 06 20:00:28.341966 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.341941    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:28.430356 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.430334    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/761d62ea-7297-4b7c-8009-28f4782541a9-maas-api-tls\") pod \"maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8\" (UID: \"761d62ea-7297-4b7c-8009-28f4782541a9\") " pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"
Jul 06 20:00:28.430451 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.430361    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/761d62ea-7297-4b7c-8009-28f4782541a9-openshift-service-ca\") pod \"maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8\" (UID: \"761d62ea-7297-4b7c-8009-28f4782541a9\") " pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"
Jul 06 20:00:28.430451 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.430395    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsq52\" (UniqueName: \"kubernetes.io/projected/761d62ea-7297-4b7c-8009-28f4782541a9-kube-api-access-tsq52\") pod \"maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8\" (UID: \"761d62ea-7297-4b7c-8009-28f4782541a9\") " pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"
Jul 06 20:00:28.430602 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:00:28.430474    2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-53be5925-serving-cert: secret "maas-api-e2e-ait-53be5925-serving-cert" not found
Jul 06 20:00:28.430602 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:00:28.430564    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/761d62ea-7297-4b7c-8009-28f4782541a9-maas-api-tls podName:761d62ea-7297-4b7c-8009-28f4782541a9 nodeName:}" failed. No retries permitted until 2026-07-06 20:00:28.930546369 +0000 UTC m=+1677.471723027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/761d62ea-7297-4b7c-8009-28f4782541a9-maas-api-tls") pod "maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8" (UID: "761d62ea-7297-4b7c-8009-28f4782541a9") : secret "maas-api-e2e-ait-53be5925-serving-cert" not found
Jul 06 20:00:28.431038 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.431017    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/761d62ea-7297-4b7c-8009-28f4782541a9-openshift-service-ca\") pod \"maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8\" (UID: \"761d62ea-7297-4b7c-8009-28f4782541a9\") " pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"
Jul 06 20:00:28.453058 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.453034    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsq52\" (UniqueName: \"kubernetes.io/projected/761d62ea-7297-4b7c-8009-28f4782541a9-kube-api-access-tsq52\") pod \"maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8\" (UID: \"761d62ea-7297-4b7c-8009-28f4782541a9\") " pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"
Jul 06 20:00:28.579175 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.579149    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:28.917507 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.917461    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:28.933064 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.933040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/761d62ea-7297-4b7c-8009-28f4782541a9-maas-api-tls\") pod \"maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8\" (UID: \"761d62ea-7297-4b7c-8009-28f4782541a9\") " pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"
Jul 06 20:00:28.935393 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:28.935376    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/761d62ea-7297-4b7c-8009-28f4782541a9-maas-api-tls\") pod \"maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8\" (UID: \"761d62ea-7297-4b7c-8009-28f4782541a9\") " pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"
Jul 06 20:00:29.019696 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:29.019674    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:29.187017 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:29.186937    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"
Jul 06 20:00:29.318430 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:29.318407    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"]
Jul 06 20:00:29.320414 ip-10-0-128-200 kubenswrapper[2562]: W0706 20:00:29.320386    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761d62ea_7297_4b7c_8009_28f4782541a9.slice/crio-765bbbf912d3200ba0d586eea7bf2598c53c875fddb7361725e6ad8d882a37d7 WatchSource:0}: Error finding container 765bbbf912d3200ba0d586eea7bf2598c53c875fddb7361725e6ad8d882a37d7: Status 404 returned error can't find the container with id 765bbbf912d3200ba0d586eea7bf2598c53c875fddb7361725e6ad8d882a37d7
Jul 06 20:00:29.321799 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:29.321778    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:00:30.082646 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:30.082616    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8" event={"ID":"761d62ea-7297-4b7c-8009-28f4782541a9","Type":"ContainerStarted","Data":"765bbbf912d3200ba0d586eea7bf2598c53c875fddb7361725e6ad8d882a37d7"}
Jul 06 20:00:30.631900 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:30.631873    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:31.088053 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:31.088020    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8" event={"ID":"761d62ea-7297-4b7c-8009-28f4782541a9","Type":"ContainerStarted","Data":"4950b4f12753eb98ed16be9c25cf817a27be100f25cb487cf6df5c54e243ac88"}
Jul 06 20:00:31.088239 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:31.088094    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"
Jul 06 20:00:31.111540 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:31.111474    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8" podStartSLOduration=1.925526903 podStartE2EDuration="3.111460443s" totalImagesPullingTime="1.18593354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:00:31.109981585 +0000 UTC m=+1679.651158262" watchObservedRunningTime="2026-07-06 20:00:31.111460443 +0000 UTC m=+1679.652637119"
Jul 06 20:00:33.024111 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:33.024078    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:33.203652 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:33.203621    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:33.288080 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:33.288013    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:33.416063 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:33.416033    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"]
Jul 06 20:00:33.416282 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:33.416240    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8" podUID="761d62ea-7297-4b7c-8009-28f4782541a9" containerName="maas-api" containerID="cri-o://4950b4f12753eb98ed16be9c25cf817a27be100f25cb487cf6df5c54e243ac88" gracePeriod=30
Jul 06 20:00:33.446458 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:33.446410    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:33.648665 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:33.648639    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"
Jul 06 20:00:33.772714 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:33.772690    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsq52\" (UniqueName: \"kubernetes.io/projected/761d62ea-7297-4b7c-8009-28f4782541a9-kube-api-access-tsq52\") pod \"761d62ea-7297-4b7c-8009-28f4782541a9\" (UID: \"761d62ea-7297-4b7c-8009-28f4782541a9\") "
Jul 06 20:00:33.772829 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:33.772764    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/761d62ea-7297-4b7c-8009-28f4782541a9-openshift-service-ca\") pod \"761d62ea-7297-4b7c-8009-28f4782541a9\" (UID: \"761d62ea-7297-4b7c-8009-28f4782541a9\") "
Jul 06 20:00:33.772829 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:33.772797    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/761d62ea-7297-4b7c-8009-28f4782541a9-maas-api-tls\") pod \"761d62ea-7297-4b7c-8009-28f4782541a9\" (UID: \"761d62ea-7297-4b7c-8009-28f4782541a9\") "
Jul 06 20:00:33.773094 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:33.773065    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761d62ea-7297-4b7c-8009-28f4782541a9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "761d62ea-7297-4b7c-8009-28f4782541a9" (UID: "761d62ea-7297-4b7c-8009-28f4782541a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:00:33.774773 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:33.774754    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761d62ea-7297-4b7c-8009-28f4782541a9-kube-api-access-tsq52" (OuterVolumeSpecName: "kube-api-access-tsq52") pod "761d62ea-7297-4b7c-8009-28f4782541a9" (UID: "761d62ea-7297-4b7c-8009-28f4782541a9"). InnerVolumeSpecName "kube-api-access-tsq52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:00:33.774831 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:33.774811    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/761d62ea-7297-4b7c-8009-28f4782541a9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "761d62ea-7297-4b7c-8009-28f4782541a9" (UID: "761d62ea-7297-4b7c-8009-28f4782541a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:00:33.874289 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:33.874237    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/761d62ea-7297-4b7c-8009-28f4782541a9-openshift-service-ca\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:00:33.874289 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:33.874258    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/761d62ea-7297-4b7c-8009-28f4782541a9-maas-api-tls\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:00:33.874289 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:33.874268    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsq52\" (UniqueName: \"kubernetes.io/projected/761d62ea-7297-4b7c-8009-28f4782541a9-kube-api-access-tsq52\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:00:34.102162 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:34.102132    2562 generic.go:356] "Generic (PLEG): container finished" podID="761d62ea-7297-4b7c-8009-28f4782541a9" containerID="4950b4f12753eb98ed16be9c25cf817a27be100f25cb487cf6df5c54e243ac88" exitCode=0
Jul 06 20:00:34.102471 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:34.102170    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8" event={"ID":"761d62ea-7297-4b7c-8009-28f4782541a9","Type":"ContainerDied","Data":"4950b4f12753eb98ed16be9c25cf817a27be100f25cb487cf6df5c54e243ac88"}
Jul 06 20:00:34.102471 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:34.102186    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"
Jul 06 20:00:34.102471 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:34.102221    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8" event={"ID":"761d62ea-7297-4b7c-8009-28f4782541a9","Type":"ContainerDied","Data":"765bbbf912d3200ba0d586eea7bf2598c53c875fddb7361725e6ad8d882a37d7"}
Jul 06 20:00:34.102471 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:34.102251    2562 scope.go:117] "RemoveContainer" containerID="4950b4f12753eb98ed16be9c25cf817a27be100f25cb487cf6df5c54e243ac88"
Jul 06 20:00:34.110869 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:34.110847    2562 scope.go:117] "RemoveContainer" containerID="4950b4f12753eb98ed16be9c25cf817a27be100f25cb487cf6df5c54e243ac88"
Jul 06 20:00:34.111198 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:00:34.111180    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4950b4f12753eb98ed16be9c25cf817a27be100f25cb487cf6df5c54e243ac88\": container with ID starting with 4950b4f12753eb98ed16be9c25cf817a27be100f25cb487cf6df5c54e243ac88 not found: ID does not exist" containerID="4950b4f12753eb98ed16be9c25cf817a27be100f25cb487cf6df5c54e243ac88"
Jul 06 20:00:34.111281 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:34.111213    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4950b4f12753eb98ed16be9c25cf817a27be100f25cb487cf6df5c54e243ac88"} err="failed to get container status \"4950b4f12753eb98ed16be9c25cf817a27be100f25cb487cf6df5c54e243ac88\": rpc error: code = NotFound desc = could not find container \"4950b4f12753eb98ed16be9c25cf817a27be100f25cb487cf6df5c54e243ac88\": container with ID starting with 4950b4f12753eb98ed16be9c25cf817a27be100f25cb487cf6df5c54e243ac88 not found: ID does not exist"
Jul 06 20:00:34.121737 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:34.121709    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"]
Jul 06 20:00:34.125737 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:34.125689    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-53be5925-865dd9fb96-q5tg8"]
Jul 06 20:00:35.895249 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:35.895207    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="761d62ea-7297-4b7c-8009-28f4782541a9" path="/var/lib/kubelet/pods/761d62ea-7297-4b7c-8009-28f4782541a9/volumes"
Jul 06 20:00:36.955824 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:36.955775    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:39.257452 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:39.257409    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:39.358778 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:39.358724    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:39.544750 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:39.544652    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:39.792480 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:39.792440    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:40.280930 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:40.280895    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:41.110598 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.110563    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:41.376696 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.376622    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"]
Jul 06 20:00:41.377098 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.377083    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="761d62ea-7297-4b7c-8009-28f4782541a9" containerName="maas-api"
Jul 06 20:00:41.377144 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.377102    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="761d62ea-7297-4b7c-8009-28f4782541a9" containerName="maas-api"
Jul 06 20:00:41.377210 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.377199    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="761d62ea-7297-4b7c-8009-28f4782541a9" containerName="maas-api"
Jul 06 20:00:41.381768 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.381749    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"
Jul 06 20:00:41.385164 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.385144    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-migrate-1dd1d29f-serving-cert\""
Jul 06 20:00:41.385277 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.385170    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-cmcxr\""
Jul 06 20:00:41.393317 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.393294    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"]
Jul 06 20:00:41.423910 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.423889    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:41.534491 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.534465    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ded2289-f5d7-4abb-bfb3-734ce50529a6-openshift-service-ca\") pod \"maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l\" (UID: \"4ded2289-f5d7-4abb-bfb3-734ce50529a6\") " pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"
Jul 06 20:00:41.534627 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.534568    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krb85\" (UniqueName: \"kubernetes.io/projected/4ded2289-f5d7-4abb-bfb3-734ce50529a6-kube-api-access-krb85\") pod \"maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l\" (UID: \"4ded2289-f5d7-4abb-bfb3-734ce50529a6\") " pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"
Jul 06 20:00:41.534627 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.534621    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ded2289-f5d7-4abb-bfb3-734ce50529a6-maas-api-tls\") pod \"maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l\" (UID: \"4ded2289-f5d7-4abb-bfb3-734ce50529a6\") " pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"
Jul 06 20:00:41.635586 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.635515    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ded2289-f5d7-4abb-bfb3-734ce50529a6-maas-api-tls\") pod \"maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l\" (UID: \"4ded2289-f5d7-4abb-bfb3-734ce50529a6\") " pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"
Jul 06 20:00:41.635586 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.635558    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ded2289-f5d7-4abb-bfb3-734ce50529a6-openshift-service-ca\") pod \"maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l\" (UID: \"4ded2289-f5d7-4abb-bfb3-734ce50529a6\") " pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"
Jul 06 20:00:41.635765 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.635628    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krb85\" (UniqueName: \"kubernetes.io/projected/4ded2289-f5d7-4abb-bfb3-734ce50529a6-kube-api-access-krb85\") pod \"maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l\" (UID: \"4ded2289-f5d7-4abb-bfb3-734ce50529a6\") " pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"
Jul 06 20:00:41.636191 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.636173    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ded2289-f5d7-4abb-bfb3-734ce50529a6-openshift-service-ca\") pod \"maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l\" (UID: \"4ded2289-f5d7-4abb-bfb3-734ce50529a6\") " pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"
Jul 06 20:00:41.637901 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.637878    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ded2289-f5d7-4abb-bfb3-734ce50529a6-maas-api-tls\") pod \"maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l\" (UID: \"4ded2289-f5d7-4abb-bfb3-734ce50529a6\") " pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"
Jul 06 20:00:41.645335 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.645317    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krb85\" (UniqueName: \"kubernetes.io/projected/4ded2289-f5d7-4abb-bfb3-734ce50529a6-kube-api-access-krb85\") pod \"maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l\" (UID: \"4ded2289-f5d7-4abb-bfb3-734ce50529a6\") " pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"
Jul 06 20:00:41.655542 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.655522    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:41.692590 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.692569    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"
Jul 06 20:00:41.825650 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.825560    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"]
Jul 06 20:00:41.827296 ip-10-0-128-200 kubenswrapper[2562]: W0706 20:00:41.827271    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ded2289_f5d7_4abb_bfb3_734ce50529a6.slice/crio-50ab124fb4bd8d9b7d683ee75e8797aa3f8e4b8c1bc06dfc145b7b826ee177bd WatchSource:0}: Error finding container 50ab124fb4bd8d9b7d683ee75e8797aa3f8e4b8c1bc06dfc145b7b826ee177bd: Status 404 returned error can't find the container with id 50ab124fb4bd8d9b7d683ee75e8797aa3f8e4b8c1bc06dfc145b7b826ee177bd
Jul 06 20:00:41.891285 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.891224    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:41.991597 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:41.991564    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:42.142203 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:42.142129    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l" event={"ID":"4ded2289-f5d7-4abb-bfb3-734ce50529a6","Type":"ContainerStarted","Data":"50ab124fb4bd8d9b7d683ee75e8797aa3f8e4b8c1bc06dfc145b7b826ee177bd"}
Jul 06 20:00:43.047860 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:43.047833    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:43.148131 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:43.148102    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l" event={"ID":"4ded2289-f5d7-4abb-bfb3-734ce50529a6","Type":"ContainerStarted","Data":"6c6cc6575e2f53c69328cfd8602327904755fc17b300413db8d92983309ebf88"}
Jul 06 20:00:43.148240 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:43.148198    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"
Jul 06 20:00:43.169085 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:43.169045    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l" podStartSLOduration=1.393470234 podStartE2EDuration="2.169031651s" totalImagesPullingTime="775.561417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:00:43.168652264 +0000 UTC m=+1691.709828939" watchObservedRunningTime="2026-07-06 20:00:43.169031651 +0000 UTC m=+1691.710208327"
Jul 06 20:00:45.584660 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:45.584620    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:45.781400 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:45.781371    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:45.889707 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:45.889632    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:46.000258 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:46.000231    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"]
Jul 06 20:00:46.000427 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:46.000404    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l" podUID="4ded2289-f5d7-4abb-bfb3-734ce50529a6" containerName="maas-api" containerID="cri-o://6c6cc6575e2f53c69328cfd8602327904755fc17b300413db8d92983309ebf88" gracePeriod=30
Jul 06 20:00:46.029730 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:46.029700    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:46.163283 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:46.163216    2562 generic.go:356] "Generic (PLEG): container finished" podID="4ded2289-f5d7-4abb-bfb3-734ce50529a6" containerID="6c6cc6575e2f53c69328cfd8602327904755fc17b300413db8d92983309ebf88" exitCode=0
Jul 06 20:00:46.163394 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:46.163288    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l" event={"ID":"4ded2289-f5d7-4abb-bfb3-734ce50529a6","Type":"ContainerDied","Data":"6c6cc6575e2f53c69328cfd8602327904755fc17b300413db8d92983309ebf88"}
Jul 06 20:00:46.249504 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:46.249459    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"
Jul 06 20:00:46.374678 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:46.374655    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ded2289-f5d7-4abb-bfb3-734ce50529a6-maas-api-tls\") pod \"4ded2289-f5d7-4abb-bfb3-734ce50529a6\" (UID: \"4ded2289-f5d7-4abb-bfb3-734ce50529a6\") "
Jul 06 20:00:46.374816 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:46.374690    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ded2289-f5d7-4abb-bfb3-734ce50529a6-openshift-service-ca\") pod \"4ded2289-f5d7-4abb-bfb3-734ce50529a6\" (UID: \"4ded2289-f5d7-4abb-bfb3-734ce50529a6\") "
Jul 06 20:00:46.374900 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:46.374882    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krb85\" (UniqueName: \"kubernetes.io/projected/4ded2289-f5d7-4abb-bfb3-734ce50529a6-kube-api-access-krb85\") pod \"4ded2289-f5d7-4abb-bfb3-734ce50529a6\" (UID: \"4ded2289-f5d7-4abb-bfb3-734ce50529a6\") "
Jul 06 20:00:46.375010 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:46.374992    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ded2289-f5d7-4abb-bfb3-734ce50529a6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4ded2289-f5d7-4abb-bfb3-734ce50529a6" (UID: "4ded2289-f5d7-4abb-bfb3-734ce50529a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:00:46.375193 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:46.375174    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ded2289-f5d7-4abb-bfb3-734ce50529a6-openshift-service-ca\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:00:46.376700 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:46.376675    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ded2289-f5d7-4abb-bfb3-734ce50529a6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4ded2289-f5d7-4abb-bfb3-734ce50529a6" (UID: "4ded2289-f5d7-4abb-bfb3-734ce50529a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:00:46.376837 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:46.376816    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ded2289-f5d7-4abb-bfb3-734ce50529a6-kube-api-access-krb85" (OuterVolumeSpecName: "kube-api-access-krb85") pod "4ded2289-f5d7-4abb-bfb3-734ce50529a6" (UID: "4ded2289-f5d7-4abb-bfb3-734ce50529a6"). InnerVolumeSpecName "kube-api-access-krb85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:00:46.475794 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:46.475743    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krb85\" (UniqueName: \"kubernetes.io/projected/4ded2289-f5d7-4abb-bfb3-734ce50529a6-kube-api-access-krb85\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:00:46.475794 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:46.475763    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ded2289-f5d7-4abb-bfb3-734ce50529a6-maas-api-tls\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:00:47.169676 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:47.169636    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l" event={"ID":"4ded2289-f5d7-4abb-bfb3-734ce50529a6","Type":"ContainerDied","Data":"50ab124fb4bd8d9b7d683ee75e8797aa3f8e4b8c1bc06dfc145b7b826ee177bd"}
Jul 06 20:00:47.170130 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:47.169703    2562 scope.go:117] "RemoveContainer" containerID="6c6cc6575e2f53c69328cfd8602327904755fc17b300413db8d92983309ebf88"
Jul 06 20:00:47.170130 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:47.169647    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"
Jul 06 20:00:47.193666 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:47.193641    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"]
Jul 06 20:00:47.198566 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:47.198547    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-migrate-1dd1d29f-5d458779b6-j5m4l"]
Jul 06 20:00:47.480080 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:47.480047    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:47.898359 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:47.898323    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ded2289-f5d7-4abb-bfb3-734ce50529a6" path="/var/lib/kubelet/pods/4ded2289-f5d7-4abb-bfb3-734ce50529a6/volumes"
Jul 06 20:00:51.926393 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:51.926356    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:52.479889 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:52.479856    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:52.580727 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:52.580697    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:53.378151 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.378120    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:53.445675 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.445634    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"]
Jul 06 20:00:53.446120 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.446104    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ded2289-f5d7-4abb-bfb3-734ce50529a6" containerName="maas-api"
Jul 06 20:00:53.446182 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.446123    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ded2289-f5d7-4abb-bfb3-734ce50529a6" containerName="maas-api"
Jul 06 20:00:53.446233 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.446220    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ded2289-f5d7-4abb-bfb3-734ce50529a6" containerName="maas-api"
Jul 06 20:00:53.450825 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.450804    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"
Jul 06 20:00:53.453717 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.453697    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-73a70dd2-serving-cert\""
Jul 06 20:00:53.453964 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.453946    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-cmcxr\""
Jul 06 20:00:53.462076 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.462051    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"]
Jul 06 20:00:53.480242 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.480215    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:53.628110 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.628084    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsxdf\" (UniqueName: \"kubernetes.io/projected/93bb4141-1690-41aa-8dc6-ad50224d3f06-kube-api-access-bsxdf\") pod \"maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p\" (UID: \"93bb4141-1690-41aa-8dc6-ad50224d3f06\") " pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"
Jul 06 20:00:53.628248 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.628114    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bb4141-1690-41aa-8dc6-ad50224d3f06-openshift-service-ca\") pod \"maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p\" (UID: \"93bb4141-1690-41aa-8dc6-ad50224d3f06\") " pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"
Jul 06 20:00:53.628248 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.628145    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93bb4141-1690-41aa-8dc6-ad50224d3f06-maas-api-tls\") pod \"maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p\" (UID: \"93bb4141-1690-41aa-8dc6-ad50224d3f06\") " pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"
Jul 06 20:00:53.729297 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.729274    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsxdf\" (UniqueName: \"kubernetes.io/projected/93bb4141-1690-41aa-8dc6-ad50224d3f06-kube-api-access-bsxdf\") pod \"maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p\" (UID: \"93bb4141-1690-41aa-8dc6-ad50224d3f06\") " pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"
Jul 06 20:00:53.729413 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.729302    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bb4141-1690-41aa-8dc6-ad50224d3f06-openshift-service-ca\") pod \"maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p\" (UID: \"93bb4141-1690-41aa-8dc6-ad50224d3f06\") " pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"
Jul 06 20:00:53.729454 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.729433    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93bb4141-1690-41aa-8dc6-ad50224d3f06-maas-api-tls\") pod \"maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p\" (UID: \"93bb4141-1690-41aa-8dc6-ad50224d3f06\") " pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"
Jul 06 20:00:53.729945 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.729929    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bb4141-1690-41aa-8dc6-ad50224d3f06-openshift-service-ca\") pod \"maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p\" (UID: \"93bb4141-1690-41aa-8dc6-ad50224d3f06\") " pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"
Jul 06 20:00:53.731810 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.731792    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93bb4141-1690-41aa-8dc6-ad50224d3f06-maas-api-tls\") pod \"maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p\" (UID: \"93bb4141-1690-41aa-8dc6-ad50224d3f06\") " pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"
Jul 06 20:00:53.739152 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.739132    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsxdf\" (UniqueName: \"kubernetes.io/projected/93bb4141-1690-41aa-8dc6-ad50224d3f06-kube-api-access-bsxdf\") pod \"maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p\" (UID: \"93bb4141-1690-41aa-8dc6-ad50224d3f06\") " pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"
Jul 06 20:00:53.764576 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.764553    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"
Jul 06 20:00:53.903386 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:53.903364    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"]
Jul 06 20:00:53.905233 ip-10-0-128-200 kubenswrapper[2562]: W0706 20:00:53.905201    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93bb4141_1690_41aa_8dc6_ad50224d3f06.slice/crio-457bf6d0ecb63f16752745b69e8b27d5cb7d8e97f8df7ddfce1208d767290ca6 WatchSource:0}: Error finding container 457bf6d0ecb63f16752745b69e8b27d5cb7d8e97f8df7ddfce1208d767290ca6: Status 404 returned error can't find the container with id 457bf6d0ecb63f16752745b69e8b27d5cb7d8e97f8df7ddfce1208d767290ca6
Jul 06 20:00:54.015639 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:54.015612    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:54.120194 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:54.120167    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:54.203220 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:54.203145    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p" event={"ID":"93bb4141-1690-41aa-8dc6-ad50224d3f06","Type":"ContainerStarted","Data":"457bf6d0ecb63f16752745b69e8b27d5cb7d8e97f8df7ddfce1208d767290ca6"}
Jul 06 20:00:55.210002 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:55.209970    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p" event={"ID":"93bb4141-1690-41aa-8dc6-ad50224d3f06","Type":"ContainerStarted","Data":"2f1be54e3902bbbc863b4723cb37efe73d0bf539a23dbea055a86c4032295bec"}
Jul 06 20:00:55.210310 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:55.210117    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"
Jul 06 20:00:55.218329 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:55.218306    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:55.233829 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:55.233793    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p" podStartSLOduration=1.366285818 podStartE2EDuration="2.233779239s" totalImagesPullingTime="867.493421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:00:55.232319909 +0000 UTC m=+1703.773496585" watchObservedRunningTime="2026-07-06 20:00:55.233779239 +0000 UTC m=+1703.774955946"
Jul 06 20:00:58.491790 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:58.491756    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:58.919475 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:58.919438    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"]
Jul 06 20:00:58.919776 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:58.919747    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p" podUID="93bb4141-1690-41aa-8dc6-ad50224d3f06" containerName="maas-api" containerID="cri-o://2f1be54e3902bbbc863b4723cb37efe73d0bf539a23dbea055a86c4032295bec" gracePeriod=30
Jul 06 20:00:59.075266 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.075228    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:00:59.168162 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.168140    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"
Jul 06 20:00:59.229611 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.229553    2562 generic.go:356] "Generic (PLEG): container finished" podID="93bb4141-1690-41aa-8dc6-ad50224d3f06" containerID="2f1be54e3902bbbc863b4723cb37efe73d0bf539a23dbea055a86c4032295bec" exitCode=0
Jul 06 20:00:59.229611 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.229594    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p" event={"ID":"93bb4141-1690-41aa-8dc6-ad50224d3f06","Type":"ContainerDied","Data":"2f1be54e3902bbbc863b4723cb37efe73d0bf539a23dbea055a86c4032295bec"}
Jul 06 20:00:59.229744 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.229631    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p" event={"ID":"93bb4141-1690-41aa-8dc6-ad50224d3f06","Type":"ContainerDied","Data":"457bf6d0ecb63f16752745b69e8b27d5cb7d8e97f8df7ddfce1208d767290ca6"}
Jul 06 20:00:59.229744 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.229651    2562 scope.go:117] "RemoveContainer" containerID="2f1be54e3902bbbc863b4723cb37efe73d0bf539a23dbea055a86c4032295bec"
Jul 06 20:00:59.229744 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.229597    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"
Jul 06 20:00:59.237400 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.237376    2562 scope.go:117] "RemoveContainer" containerID="2f1be54e3902bbbc863b4723cb37efe73d0bf539a23dbea055a86c4032295bec"
Jul 06 20:00:59.237687 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:00:59.237668    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f1be54e3902bbbc863b4723cb37efe73d0bf539a23dbea055a86c4032295bec\": container with ID starting with 2f1be54e3902bbbc863b4723cb37efe73d0bf539a23dbea055a86c4032295bec not found: ID does not exist" containerID="2f1be54e3902bbbc863b4723cb37efe73d0bf539a23dbea055a86c4032295bec"
Jul 06 20:00:59.237758 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.237701    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f1be54e3902bbbc863b4723cb37efe73d0bf539a23dbea055a86c4032295bec"} err="failed to get container status \"2f1be54e3902bbbc863b4723cb37efe73d0bf539a23dbea055a86c4032295bec\": rpc error: code = NotFound desc = could not find container \"2f1be54e3902bbbc863b4723cb37efe73d0bf539a23dbea055a86c4032295bec\": container with ID starting with 2f1be54e3902bbbc863b4723cb37efe73d0bf539a23dbea055a86c4032295bec not found: ID does not exist"
Jul 06 20:00:59.274645 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.274624    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsxdf\" (UniqueName: \"kubernetes.io/projected/93bb4141-1690-41aa-8dc6-ad50224d3f06-kube-api-access-bsxdf\") pod \"93bb4141-1690-41aa-8dc6-ad50224d3f06\" (UID: \"93bb4141-1690-41aa-8dc6-ad50224d3f06\") "
Jul 06 20:00:59.274739 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.274727    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bb4141-1690-41aa-8dc6-ad50224d3f06-openshift-service-ca\") pod \"93bb4141-1690-41aa-8dc6-ad50224d3f06\" (UID: \"93bb4141-1690-41aa-8dc6-ad50224d3f06\") "
Jul 06 20:00:59.274775 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.274747    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93bb4141-1690-41aa-8dc6-ad50224d3f06-maas-api-tls\") pod \"93bb4141-1690-41aa-8dc6-ad50224d3f06\" (UID: \"93bb4141-1690-41aa-8dc6-ad50224d3f06\") "
Jul 06 20:00:59.275015 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.274994    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93bb4141-1690-41aa-8dc6-ad50224d3f06-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "93bb4141-1690-41aa-8dc6-ad50224d3f06" (UID: "93bb4141-1690-41aa-8dc6-ad50224d3f06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:00:59.276796 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.276774    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93bb4141-1690-41aa-8dc6-ad50224d3f06-kube-api-access-bsxdf" (OuterVolumeSpecName: "kube-api-access-bsxdf") pod "93bb4141-1690-41aa-8dc6-ad50224d3f06" (UID: "93bb4141-1690-41aa-8dc6-ad50224d3f06"). InnerVolumeSpecName "kube-api-access-bsxdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:00:59.276875 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.276775    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93bb4141-1690-41aa-8dc6-ad50224d3f06-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "93bb4141-1690-41aa-8dc6-ad50224d3f06" (UID: "93bb4141-1690-41aa-8dc6-ad50224d3f06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:00:59.375642 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.375618    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bb4141-1690-41aa-8dc6-ad50224d3f06-openshift-service-ca\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:00:59.375642 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.375639    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93bb4141-1690-41aa-8dc6-ad50224d3f06-maas-api-tls\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:00:59.375756 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.375648    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsxdf\" (UniqueName: \"kubernetes.io/projected/93bb4141-1690-41aa-8dc6-ad50224d3f06-kube-api-access-bsxdf\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:00:59.563877 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.563853    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"]
Jul 06 20:00:59.578396 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.578373    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-73a70dd2-f75b69878-9hn2p"]
Jul 06 20:00:59.894918 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:00:59.894855    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93bb4141-1690-41aa-8dc6-ad50224d3f06" path="/var/lib/kubelet/pods/93bb4141-1690-41aa-8dc6-ad50224d3f06/volumes"
Jul 06 20:01:04.512818 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:04.512786    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:04.622459 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:04.622429    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:06.923287 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:06.923254    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:10.592416 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:10.592387    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:10.694963 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:10.694926    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:10.813003 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:10.812975    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:10.927419 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:10.927334    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:12.009316 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.009271    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:12.084301 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.084272    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"]
Jul 06 20:01:12.084649 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.084632    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93bb4141-1690-41aa-8dc6-ad50224d3f06" containerName="maas-api"
Jul 06 20:01:12.084649 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.084648    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="93bb4141-1690-41aa-8dc6-ad50224d3f06" containerName="maas-api"
Jul 06 20:01:12.084782 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.084731    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="93bb4141-1690-41aa-8dc6-ad50224d3f06" containerName="maas-api"
Jul 06 20:01:12.087770 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.087753    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"
Jul 06 20:01:12.091321 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.091302    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-cmcxr\""
Jul 06 20:01:12.091414 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.091321    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-0dc8851b-serving-cert\""
Jul 06 20:01:12.108528 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.108507    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"]
Jul 06 20:01:12.166363 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.166327    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07587450-4a95-4f2a-801b-6f269b98beef-openshift-service-ca\") pod \"maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz\" (UID: \"07587450-4a95-4f2a-801b-6f269b98beef\") " pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"
Jul 06 20:01:12.166498 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.166400    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07587450-4a95-4f2a-801b-6f269b98beef-maas-api-tls\") pod \"maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz\" (UID: \"07587450-4a95-4f2a-801b-6f269b98beef\") " pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"
Jul 06 20:01:12.166543 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.166524    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4459s\" (UniqueName: \"kubernetes.io/projected/07587450-4a95-4f2a-801b-6f269b98beef-kube-api-access-4459s\") pod \"maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz\" (UID: \"07587450-4a95-4f2a-801b-6f269b98beef\") " pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"
Jul 06 20:01:12.227186 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.227152    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:12.266856 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.266797    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4459s\" (UniqueName: \"kubernetes.io/projected/07587450-4a95-4f2a-801b-6f269b98beef-kube-api-access-4459s\") pod \"maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz\" (UID: \"07587450-4a95-4f2a-801b-6f269b98beef\") " pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"
Jul 06 20:01:12.266856 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.266828    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07587450-4a95-4f2a-801b-6f269b98beef-openshift-service-ca\") pod \"maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz\" (UID: \"07587450-4a95-4f2a-801b-6f269b98beef\") " pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"
Jul 06 20:01:12.266856 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.266852    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07587450-4a95-4f2a-801b-6f269b98beef-maas-api-tls\") pod \"maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz\" (UID: \"07587450-4a95-4f2a-801b-6f269b98beef\") " pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"
Jul 06 20:01:12.267444 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.267422    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07587450-4a95-4f2a-801b-6f269b98beef-openshift-service-ca\") pod \"maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz\" (UID: \"07587450-4a95-4f2a-801b-6f269b98beef\") " pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"
Jul 06 20:01:12.269088 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.269067    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07587450-4a95-4f2a-801b-6f269b98beef-maas-api-tls\") pod \"maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz\" (UID: \"07587450-4a95-4f2a-801b-6f269b98beef\") " pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"
Jul 06 20:01:12.280339 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.280312    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4459s\" (UniqueName: \"kubernetes.io/projected/07587450-4a95-4f2a-801b-6f269b98beef-kube-api-access-4459s\") pod \"maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz\" (UID: \"07587450-4a95-4f2a-801b-6f269b98beef\") " pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"
Jul 06 20:01:12.325588 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.325561    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:12.398456 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.398431    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"
Jul 06 20:01:12.535908 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.535887    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"]
Jul 06 20:01:12.537866 ip-10-0-128-200 kubenswrapper[2562]: W0706 20:01:12.537839    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07587450_4a95_4f2a_801b_6f269b98beef.slice/crio-e23f535dbbfb56e84e255b442ac952c2e97828a720f7751ac025db153cd92bba WatchSource:0}: Error finding container e23f535dbbfb56e84e255b442ac952c2e97828a720f7751ac025db153cd92bba: Status 404 returned error can't find the container with id e23f535dbbfb56e84e255b442ac952c2e97828a720f7751ac025db153cd92bba
Jul 06 20:01:12.665272 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.665246    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:12.767586 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:12.767561    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:13.293552 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:13.293522    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz" event={"ID":"07587450-4a95-4f2a-801b-6f269b98beef","Type":"ContainerStarted","Data":"e23f535dbbfb56e84e255b442ac952c2e97828a720f7751ac025db153cd92bba"}
Jul 06 20:01:14.294444 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:14.294415    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:14.299364 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:14.299339    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz" event={"ID":"07587450-4a95-4f2a-801b-6f269b98beef","Type":"ContainerStarted","Data":"53a8e71c640c7d65003cfe6e1fc06d873946aa60ec1c0e81ca149129da833964"}
Jul 06 20:01:14.299473 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:14.299442    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"
Jul 06 20:01:14.334403 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:14.334358    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz" podStartSLOduration=1.410990233 podStartE2EDuration="2.334342318s" totalImagesPullingTime="923.352085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:01:14.333143246 +0000 UTC m=+1722.874319928" watchObservedRunningTime="2026-07-06 20:01:14.334342318 +0000 UTC m=+1722.875518999"
Jul 06 20:01:16.359105 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:16.359056    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:16.561696 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:16.561663    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:16.659597 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:16.659476    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:16.742534 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:16.742506    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"]
Jul 06 20:01:16.742749 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:16.742717    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz" podUID="07587450-4a95-4f2a-801b-6f269b98beef" containerName="maas-api" containerID="cri-o://53a8e71c640c7d65003cfe6e1fc06d873946aa60ec1c0e81ca149129da833964" gracePeriod=30
Jul 06 20:01:16.783634 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:16.783598    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:16.992387 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:16.992365    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"
Jul 06 20:01:16.999572 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:16.999551    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07587450-4a95-4f2a-801b-6f269b98beef-maas-api-tls\") pod \"07587450-4a95-4f2a-801b-6f269b98beef\" (UID: \"07587450-4a95-4f2a-801b-6f269b98beef\") "
Jul 06 20:01:16.999678 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:16.999618    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4459s\" (UniqueName: \"kubernetes.io/projected/07587450-4a95-4f2a-801b-6f269b98beef-kube-api-access-4459s\") pod \"07587450-4a95-4f2a-801b-6f269b98beef\" (UID: \"07587450-4a95-4f2a-801b-6f269b98beef\") "
Jul 06 20:01:16.999745 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:16.999730    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07587450-4a95-4f2a-801b-6f269b98beef-openshift-service-ca\") pod \"07587450-4a95-4f2a-801b-6f269b98beef\" (UID: \"07587450-4a95-4f2a-801b-6f269b98beef\") "
Jul 06 20:01:17.000030 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:17.000007    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07587450-4a95-4f2a-801b-6f269b98beef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "07587450-4a95-4f2a-801b-6f269b98beef" (UID: "07587450-4a95-4f2a-801b-6f269b98beef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:01:17.001453 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:17.001433    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07587450-4a95-4f2a-801b-6f269b98beef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "07587450-4a95-4f2a-801b-6f269b98beef" (UID: "07587450-4a95-4f2a-801b-6f269b98beef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:01:17.001541 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:17.001474    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07587450-4a95-4f2a-801b-6f269b98beef-kube-api-access-4459s" (OuterVolumeSpecName: "kube-api-access-4459s") pod "07587450-4a95-4f2a-801b-6f269b98beef" (UID: "07587450-4a95-4f2a-801b-6f269b98beef"). InnerVolumeSpecName "kube-api-access-4459s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:01:17.100754 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:17.100728    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07587450-4a95-4f2a-801b-6f269b98beef-maas-api-tls\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:01:17.100754 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:17.100750    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4459s\" (UniqueName: \"kubernetes.io/projected/07587450-4a95-4f2a-801b-6f269b98beef-kube-api-access-4459s\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:01:17.100754 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:17.100759    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07587450-4a95-4f2a-801b-6f269b98beef-openshift-service-ca\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:01:17.315528 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:17.315498    2562 generic.go:356] "Generic (PLEG): container finished" podID="07587450-4a95-4f2a-801b-6f269b98beef" containerID="53a8e71c640c7d65003cfe6e1fc06d873946aa60ec1c0e81ca149129da833964" exitCode=0
Jul 06 20:01:17.315643 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:17.315540    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"
Jul 06 20:01:17.315643 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:17.315563    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz" event={"ID":"07587450-4a95-4f2a-801b-6f269b98beef","Type":"ContainerDied","Data":"53a8e71c640c7d65003cfe6e1fc06d873946aa60ec1c0e81ca149129da833964"}
Jul 06 20:01:17.315643 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:17.315623    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz" event={"ID":"07587450-4a95-4f2a-801b-6f269b98beef","Type":"ContainerDied","Data":"e23f535dbbfb56e84e255b442ac952c2e97828a720f7751ac025db153cd92bba"}
Jul 06 20:01:17.315805 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:17.315657    2562 scope.go:117] "RemoveContainer" containerID="53a8e71c640c7d65003cfe6e1fc06d873946aa60ec1c0e81ca149129da833964"
Jul 06 20:01:17.325318 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:17.325298    2562 scope.go:117] "RemoveContainer" containerID="53a8e71c640c7d65003cfe6e1fc06d873946aa60ec1c0e81ca149129da833964"
Jul 06 20:01:17.325608 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:01:17.325582    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53a8e71c640c7d65003cfe6e1fc06d873946aa60ec1c0e81ca149129da833964\": container with ID starting with 53a8e71c640c7d65003cfe6e1fc06d873946aa60ec1c0e81ca149129da833964 not found: ID does not exist" containerID="53a8e71c640c7d65003cfe6e1fc06d873946aa60ec1c0e81ca149129da833964"
Jul 06 20:01:17.325661 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:17.325618    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a8e71c640c7d65003cfe6e1fc06d873946aa60ec1c0e81ca149129da833964"} err="failed to get container status \"53a8e71c640c7d65003cfe6e1fc06d873946aa60ec1c0e81ca149129da833964\": rpc error: code = NotFound desc = could not find container \"53a8e71c640c7d65003cfe6e1fc06d873946aa60ec1c0e81ca149129da833964\": container with ID starting with 53a8e71c640c7d65003cfe6e1fc06d873946aa60ec1c0e81ca149129da833964 not found: ID does not exist"
Jul 06 20:01:17.342806 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:17.342780    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"]
Jul 06 20:01:17.347686 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:17.347666    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-0dc8851b-67bd788bd4-h9vmz"]
Jul 06 20:01:17.895291 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:17.895255    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07587450-4a95-4f2a-801b-6f269b98beef" path="/var/lib/kubelet/pods/07587450-4a95-4f2a-801b-6f269b98beef/volumes"
Jul 06 20:01:18.268145 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:18.268118    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:23.420497 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:23.420461    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:23.519329 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:23.519301    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:24.965581 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:24.965542    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:31.628613 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:31.628571    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:31.729397 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:31.729361    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:31.862651 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:31.862624    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:32.963886 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:32.963849    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:01:57.831930 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:01:57.831891    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:04.198220 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:04.198183    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:04.390276 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:04.390242    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:04.521694 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:04.521664    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:20.293096 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:20.293031    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:26.394317 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:26.394186    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:26.499300 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:26.499264    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:26.613862 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:26.613830    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:37.119144 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:37.119110    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:37.924016 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:37.923984    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:41.801661 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:41.801622    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:43.919029 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:43.918992    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:44.207185 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:44.207082    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:44.307977 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:44.307946    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:44.956598 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:44.956568    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:45.051735 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:45.051695    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:45.151866 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:45.151839    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:46.761255 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:46.761223    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:53.192531 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:53.192479    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:53.291556 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:53.291527    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:53.868915 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:53.868877    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:02:59.777844 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:02:59.777804    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:01.127283 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:01.127248    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:03.052874 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.052829    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:03.166388 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.166343    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:03.198226 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.198197    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"]
Jul 06 20:03:03.198541 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.198528    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07587450-4a95-4f2a-801b-6f269b98beef" containerName="maas-api"
Jul 06 20:03:03.198748 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.198543    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="07587450-4a95-4f2a-801b-6f269b98beef" containerName="maas-api"
Jul 06 20:03:03.198748 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.198626    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="07587450-4a95-4f2a-801b-6f269b98beef" containerName="maas-api"
Jul 06 20:03:03.201580 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.201561    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"
Jul 06 20:03:03.205011 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.204989    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-adfb9455-serving-cert\""
Jul 06 20:03:03.205108 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.205020    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-cmcxr\""
Jul 06 20:03:03.220234 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.220213    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"]
Jul 06 20:03:03.286008 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.285983    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-openshift-service-ca\") pod \"maas-api-e2e-mt-adfb9455-667ccb7985-sptc5\" (UID: \"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc\") " pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"
Jul 06 20:03:03.286141 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.286032    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-maas-api-tls\") pod \"maas-api-e2e-mt-adfb9455-667ccb7985-sptc5\" (UID: \"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc\") " pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"
Jul 06 20:03:03.286141 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.286060    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r748\" (UniqueName: \"kubernetes.io/projected/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-kube-api-access-5r748\") pod \"maas-api-e2e-mt-adfb9455-667ccb7985-sptc5\" (UID: \"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc\") " pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"
Jul 06 20:03:03.305267 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.305203    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:03.386349 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.386322    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-openshift-service-ca\") pod \"maas-api-e2e-mt-adfb9455-667ccb7985-sptc5\" (UID: \"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc\") " pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"
Jul 06 20:03:03.386457 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.386363    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-maas-api-tls\") pod \"maas-api-e2e-mt-adfb9455-667ccb7985-sptc5\" (UID: \"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc\") " pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"
Jul 06 20:03:03.386457 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.386385    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r748\" (UniqueName: \"kubernetes.io/projected/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-kube-api-access-5r748\") pod \"maas-api-e2e-mt-adfb9455-667ccb7985-sptc5\" (UID: \"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc\") " pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"
Jul 06 20:03:03.386571 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:03:03.386480    2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-adfb9455-serving-cert: secret "maas-api-e2e-mt-adfb9455-serving-cert" not found
Jul 06 20:03:03.386608 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:03:03.386593    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-maas-api-tls podName:f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc nodeName:}" failed. No retries permitted until 2026-07-06 20:03:03.886561554 +0000 UTC m=+1832.427738207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-maas-api-tls") pod "maas-api-e2e-mt-adfb9455-667ccb7985-sptc5" (UID: "f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc") : secret "maas-api-e2e-mt-adfb9455-serving-cert" not found
Jul 06 20:03:03.387033 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.387014    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-openshift-service-ca\") pod \"maas-api-e2e-mt-adfb9455-667ccb7985-sptc5\" (UID: \"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc\") " pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"
Jul 06 20:03:03.398424 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.398400    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r748\" (UniqueName: \"kubernetes.io/projected/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-kube-api-access-5r748\") pod \"maas-api-e2e-mt-adfb9455-667ccb7985-sptc5\" (UID: \"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc\") " pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"
Jul 06 20:03:03.631905 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.631841    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:03.860400 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.860372    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:03.888955 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.888897    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-maas-api-tls\") pod \"maas-api-e2e-mt-adfb9455-667ccb7985-sptc5\" (UID: \"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc\") " pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"
Jul 06 20:03:03.891413 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.891389    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-maas-api-tls\") pod \"maas-api-e2e-mt-adfb9455-667ccb7985-sptc5\" (UID: \"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc\") " pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"
Jul 06 20:03:03.989352 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:03.989328    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:04.110978 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:04.110939    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"
Jul 06 20:03:04.249964 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:04.249944    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"]
Jul 06 20:03:04.251677 ip-10-0-128-200 kubenswrapper[2562]: W0706 20:03:04.251647    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf698e9e1_cb3c_4a7b_9241_db6cf1a44ccc.slice/crio-637d3d431d58e8a99b323ab01963c67815ef683b07b5b7a42bb19194892ebfb9 WatchSource:0}: Error finding container 637d3d431d58e8a99b323ab01963c67815ef683b07b5b7a42bb19194892ebfb9: Status 404 returned error can't find the container with id 637d3d431d58e8a99b323ab01963c67815ef683b07b5b7a42bb19194892ebfb9
Jul 06 20:03:04.814880 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:04.814840    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5" event={"ID":"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc","Type":"ContainerStarted","Data":"637d3d431d58e8a99b323ab01963c67815ef683b07b5b7a42bb19194892ebfb9"}
Jul 06 20:03:05.820570 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:05.820532    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5" event={"ID":"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc","Type":"ContainerStarted","Data":"f54b53c9726bb2c4345a7f12a4921bb44c75d0403ad37d5c7b6c4b8e1a168741"}
Jul 06 20:03:05.820991 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:05.820731    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"
Jul 06 20:03:05.871695 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:05.871646    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5" podStartSLOduration=2.256745771 podStartE2EDuration="2.871629569s" totalImagesPullingTime="614.883798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:03:05.87032352 +0000 UTC m=+1834.411500198" watchObservedRunningTime="2026-07-06 20:03:05.871629569 +0000 UTC m=+1834.412806245"
Jul 06 20:03:07.576665 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:07.576602    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:08.267690 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:08.267642    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:08.432045 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:08.432016    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:08.695097 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:08.695015    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:11.829972 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:11.829943    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"
Jul 06 20:03:14.689740 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:14.689704    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:14.898319 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:14.898286    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:15.123794 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.123742    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:15.461419 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.461345    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"]
Jul 06 20:03:15.461625 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.461599    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5" podUID="f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc" containerName="maas-api" containerID="cri-o://f54b53c9726bb2c4345a7f12a4921bb44c75d0403ad37d5c7b6c4b8e1a168741" gracePeriod=30
Jul 06 20:03:15.491680 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.491655    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:15.710228 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.710204    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"
Jul 06 20:03:15.733431 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.733369    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:15.773590 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.773568    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-maas-api-tls\") pod \"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc\" (UID: \"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc\") "
Jul 06 20:03:15.773685 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.773608    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-openshift-service-ca\") pod \"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc\" (UID: \"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc\") "
Jul 06 20:03:15.773685 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.773667    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r748\" (UniqueName: \"kubernetes.io/projected/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-kube-api-access-5r748\") pod \"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc\" (UID: \"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc\") "
Jul 06 20:03:15.773978 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.773953    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc" (UID: "f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:03:15.775593 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.775571    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc" (UID: "f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:03:15.775681 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.775603    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-kube-api-access-5r748" (OuterVolumeSpecName: "kube-api-access-5r748") pod "f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc" (UID: "f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc"). InnerVolumeSpecName "kube-api-access-5r748". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:03:15.866650 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.866623    2562 generic.go:356] "Generic (PLEG): container finished" podID="f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc" containerID="f54b53c9726bb2c4345a7f12a4921bb44c75d0403ad37d5c7b6c4b8e1a168741" exitCode=0
Jul 06 20:03:15.866754 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.866656    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5" event={"ID":"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc","Type":"ContainerDied","Data":"f54b53c9726bb2c4345a7f12a4921bb44c75d0403ad37d5c7b6c4b8e1a168741"}
Jul 06 20:03:15.866754 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.866675    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"
Jul 06 20:03:15.866754 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.866738    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5" event={"ID":"f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc","Type":"ContainerDied","Data":"637d3d431d58e8a99b323ab01963c67815ef683b07b5b7a42bb19194892ebfb9"}
Jul 06 20:03:15.866918 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.866774    2562 scope.go:117] "RemoveContainer" containerID="f54b53c9726bb2c4345a7f12a4921bb44c75d0403ad37d5c7b6c4b8e1a168741"
Jul 06 20:03:15.874200 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.874154    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-maas-api-tls\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:03:15.874200 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.874178    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-openshift-service-ca\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:03:15.874200 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.874191    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r748\" (UniqueName: \"kubernetes.io/projected/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc-kube-api-access-5r748\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:03:15.875151 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.875135    2562 scope.go:117] "RemoveContainer" containerID="f54b53c9726bb2c4345a7f12a4921bb44c75d0403ad37d5c7b6c4b8e1a168741"
Jul 06 20:03:15.875401 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:03:15.875382    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f54b53c9726bb2c4345a7f12a4921bb44c75d0403ad37d5c7b6c4b8e1a168741\": container with ID starting with f54b53c9726bb2c4345a7f12a4921bb44c75d0403ad37d5c7b6c4b8e1a168741 not found: ID does not exist" containerID="f54b53c9726bb2c4345a7f12a4921bb44c75d0403ad37d5c7b6c4b8e1a168741"
Jul 06 20:03:15.875460 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.875410    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f54b53c9726bb2c4345a7f12a4921bb44c75d0403ad37d5c7b6c4b8e1a168741"} err="failed to get container status \"f54b53c9726bb2c4345a7f12a4921bb44c75d0403ad37d5c7b6c4b8e1a168741\": rpc error: code = NotFound desc = could not find container \"f54b53c9726bb2c4345a7f12a4921bb44c75d0403ad37d5c7b6c4b8e1a168741\": container with ID starting with f54b53c9726bb2c4345a7f12a4921bb44c75d0403ad37d5c7b6c4b8e1a168741 not found: ID does not exist"
Jul 06 20:03:15.887970 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.887949    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"]
Jul 06 20:03:15.895535 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:15.895518    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-adfb9455-667ccb7985-sptc5"]
Jul 06 20:03:17.894350 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:17.894317    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc" path="/var/lib/kubelet/pods/f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc/volumes"
Jul 06 20:03:21.484154 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:21.484106    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:21.581262 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:21.581228    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:21.731207 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:21.731178    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:21.946302 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:21.946271    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:22.073824 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:22.073781    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:22.162403 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:22.162377    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:22.283327 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:22.283306    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:28.295633 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:28.295599    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:28.492825 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:28.492785    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:28.652072 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:28.651993    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:28.935822 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:28.935734    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:30.556530 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:30.556499    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:31.083649 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:31.083615    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:34.562421 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:34.562383    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:34.693294 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:34.693260    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:41.380362 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:41.380300    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:41.612387 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:41.612356    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:41.886676 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:41.886626    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:44.245765 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:44.245723    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:44.502969 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:44.502939    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"]
Jul 06 20:03:44.503264 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:44.503250    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc" containerName="maas-api"
Jul 06 20:03:44.503264 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:44.503264    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc" containerName="maas-api"
Jul 06 20:03:44.503397 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:44.503323    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f698e9e1-cb3c-4a7b-9241-db6cf1a44ccc" containerName="maas-api"
Jul 06 20:03:44.506258 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:44.506241    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"
Jul 06 20:03:44.509549 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:44.509524    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-84fa8f39-serving-cert\""
Jul 06 20:03:44.509674 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:44.509563    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-cmcxr\""
Jul 06 20:03:44.524383 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:44.524361    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:44.526219 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:44.526200    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"]
Jul 06 20:03:44.600567 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:44.600543    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9db4314-247f-4a6f-a1dd-3d970f254277-maas-api-tls\") pod \"maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x\" (UID: \"b9db4314-247f-4a6f-a1dd-3d970f254277\") " pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"
Jul 06 20:03:44.600689 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:44.600601    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t528t\" (UniqueName: \"kubernetes.io/projected/b9db4314-247f-4a6f-a1dd-3d970f254277-kube-api-access-t528t\") pod \"maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x\" (UID: \"b9db4314-247f-4a6f-a1dd-3d970f254277\") " pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"
Jul 06 20:03:44.600689 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:44.600625    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9db4314-247f-4a6f-a1dd-3d970f254277-openshift-service-ca\") pod \"maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x\" (UID: \"b9db4314-247f-4a6f-a1dd-3d970f254277\") " pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"
Jul 06 20:03:44.616108 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:44.616087    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:44.701327 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:44.701306    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9db4314-247f-4a6f-a1dd-3d970f254277-maas-api-tls\") pod \"maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x\" (UID: \"b9db4314-247f-4a6f-a1dd-3d970f254277\") " pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"
Jul 06 20:03:44.701438 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:44.701356    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t528t\" (UniqueName: \"kubernetes.io/projected/b9db4314-247f-4a6f-a1dd-3d970f254277-kube-api-access-t528t\") pod \"maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x\" (UID: \"b9db4314-247f-4a6f-a1dd-3d970f254277\") " pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"
Jul 06 20:03:44.701438 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:44.701375    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9db4314-247f-4a6f-a1dd-3d970f254277-openshift-service-ca\") pod \"maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x\" (UID: \"b9db4314-247f-4a6f-a1dd-3d970f254277\") " pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"
Jul 06 20:03:44.701589 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:03:44.701471    2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-84fa8f39-serving-cert: secret "maas-api-e2e-mt-84fa8f39-serving-cert" not found
Jul 06 20:03:44.701589 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:03:44.701564    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9db4314-247f-4a6f-a1dd-3d970f254277-maas-api-tls podName:b9db4314-247f-4a6f-a1dd-3d970f254277 nodeName:}" failed. No retries permitted until 2026-07-06 20:03:45.201543419 +0000 UTC m=+1873.742720074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9db4314-247f-4a6f-a1dd-3d970f254277-maas-api-tls") pod "maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x" (UID: "b9db4314-247f-4a6f-a1dd-3d970f254277") : secret "maas-api-e2e-mt-84fa8f39-serving-cert" not found
Jul 06 20:03:44.701948 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:44.701929    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9db4314-247f-4a6f-a1dd-3d970f254277-openshift-service-ca\") pod \"maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x\" (UID: \"b9db4314-247f-4a6f-a1dd-3d970f254277\") " pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"
Jul 06 20:03:44.710512 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:44.710472    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t528t\" (UniqueName: \"kubernetes.io/projected/b9db4314-247f-4a6f-a1dd-3d970f254277-kube-api-access-t528t\") pod \"maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x\" (UID: \"b9db4314-247f-4a6f-a1dd-3d970f254277\") " pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"
Jul 06 20:03:45.060920 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:45.060894    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:45.204742 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:45.204712    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9db4314-247f-4a6f-a1dd-3d970f254277-maas-api-tls\") pod \"maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x\" (UID: \"b9db4314-247f-4a6f-a1dd-3d970f254277\") " pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"
Jul 06 20:03:45.206975 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:45.206954    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9db4314-247f-4a6f-a1dd-3d970f254277-maas-api-tls\") pod \"maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x\" (UID: \"b9db4314-247f-4a6f-a1dd-3d970f254277\") " pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"
Jul 06 20:03:45.238252 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:45.238226    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:45.337701 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:45.337642    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:45.417134 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:45.417103    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"
Jul 06 20:03:45.550130 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:45.550098    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"]
Jul 06 20:03:45.552266 ip-10-0-128-200 kubenswrapper[2562]: W0706 20:03:45.552229    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9db4314_247f_4a6f_a1dd_3d970f254277.slice/crio-dd020380068723efb48597a23a1dc5bed28bb0647569bb6f606051d24b109336 WatchSource:0}: Error finding container dd020380068723efb48597a23a1dc5bed28bb0647569bb6f606051d24b109336: Status 404 returned error can't find the container with id dd020380068723efb48597a23a1dc5bed28bb0647569bb6f606051d24b109336
Jul 06 20:03:46.011376 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:46.011350    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x" event={"ID":"b9db4314-247f-4a6f-a1dd-3d970f254277","Type":"ContainerStarted","Data":"dd020380068723efb48597a23a1dc5bed28bb0647569bb6f606051d24b109336"}
Jul 06 20:03:47.018595 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:47.018553    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x" event={"ID":"b9db4314-247f-4a6f-a1dd-3d970f254277","Type":"ContainerStarted","Data":"22ecb9ba5c13b5a527c88784e63a07c01ce012e47b1f9eacabf87c056424e802"}
Jul 06 20:03:47.019017 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:47.018627    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"
Jul 06 20:03:47.043324 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:47.043268    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x" podStartSLOduration=2.615524472 podStartE2EDuration="3.043251138s" totalImagesPullingTime="427.726666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:03:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:03:47.04054412 +0000 UTC m=+1875.581720797" watchObservedRunningTime="2026-07-06 20:03:47.043251138 +0000 UTC m=+1875.584427814"
Jul 06 20:03:48.800536 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:48.800496    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:53.029003 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:53.028967    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"
Jul 06 20:03:58.327959 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:58.327878    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:03:58.537873 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:03:58.537845    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:04:00.947623 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:00.947589    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:04:36.476250 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:36.476205    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:04:36.823260 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:36.823223    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"]
Jul 06 20:04:36.823537 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:36.823507    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x" podUID="b9db4314-247f-4a6f-a1dd-3d970f254277" containerName="maas-api" containerID="cri-o://22ecb9ba5c13b5a527c88784e63a07c01ce012e47b1f9eacabf87c056424e802" gracePeriod=30
Jul 06 20:04:36.936128 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:36.936103    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:04:37.074032 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.073974    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"
Jul 06 20:04:37.080820 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.080796    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9db4314-247f-4a6f-a1dd-3d970f254277-openshift-service-ca\") pod \"b9db4314-247f-4a6f-a1dd-3d970f254277\" (UID: \"b9db4314-247f-4a6f-a1dd-3d970f254277\") "
Jul 06 20:04:37.080905 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.080890    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9db4314-247f-4a6f-a1dd-3d970f254277-maas-api-tls\") pod \"b9db4314-247f-4a6f-a1dd-3d970f254277\" (UID: \"b9db4314-247f-4a6f-a1dd-3d970f254277\") "
Jul 06 20:04:37.080974 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.080960    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t528t\" (UniqueName: \"kubernetes.io/projected/b9db4314-247f-4a6f-a1dd-3d970f254277-kube-api-access-t528t\") pod \"b9db4314-247f-4a6f-a1dd-3d970f254277\" (UID: \"b9db4314-247f-4a6f-a1dd-3d970f254277\") "
Jul 06 20:04:37.081173 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.081157    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9db4314-247f-4a6f-a1dd-3d970f254277-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b9db4314-247f-4a6f-a1dd-3d970f254277" (UID: "b9db4314-247f-4a6f-a1dd-3d970f254277"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:04:37.082807 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.082778    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9db4314-247f-4a6f-a1dd-3d970f254277-kube-api-access-t528t" (OuterVolumeSpecName: "kube-api-access-t528t") pod "b9db4314-247f-4a6f-a1dd-3d970f254277" (UID: "b9db4314-247f-4a6f-a1dd-3d970f254277"). InnerVolumeSpecName "kube-api-access-t528t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:04:37.082905 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.082868    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9db4314-247f-4a6f-a1dd-3d970f254277-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b9db4314-247f-4a6f-a1dd-3d970f254277" (UID: "b9db4314-247f-4a6f-a1dd-3d970f254277"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:04:37.182312 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.182282    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9db4314-247f-4a6f-a1dd-3d970f254277-maas-api-tls\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:04:37.182312 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.182306    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t528t\" (UniqueName: \"kubernetes.io/projected/b9db4314-247f-4a6f-a1dd-3d970f254277-kube-api-access-t528t\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:04:37.182312 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.182316    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9db4314-247f-4a6f-a1dd-3d970f254277-openshift-service-ca\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:04:37.245419 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.245387    2562 generic.go:356] "Generic (PLEG): container finished" podID="b9db4314-247f-4a6f-a1dd-3d970f254277" containerID="22ecb9ba5c13b5a527c88784e63a07c01ce012e47b1f9eacabf87c056424e802" exitCode=0
Jul 06 20:04:37.245587 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.245433    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"
Jul 06 20:04:37.245587 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.245467    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x" event={"ID":"b9db4314-247f-4a6f-a1dd-3d970f254277","Type":"ContainerDied","Data":"22ecb9ba5c13b5a527c88784e63a07c01ce012e47b1f9eacabf87c056424e802"}
Jul 06 20:04:37.245587 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.245551    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x" event={"ID":"b9db4314-247f-4a6f-a1dd-3d970f254277","Type":"ContainerDied","Data":"dd020380068723efb48597a23a1dc5bed28bb0647569bb6f606051d24b109336"}
Jul 06 20:04:37.245587 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.245582    2562 scope.go:117] "RemoveContainer" containerID="22ecb9ba5c13b5a527c88784e63a07c01ce012e47b1f9eacabf87c056424e802"
Jul 06 20:04:37.255642 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.255625    2562 scope.go:117] "RemoveContainer" containerID="22ecb9ba5c13b5a527c88784e63a07c01ce012e47b1f9eacabf87c056424e802"
Jul 06 20:04:37.255908 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:04:37.255888    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ecb9ba5c13b5a527c88784e63a07c01ce012e47b1f9eacabf87c056424e802\": container with ID starting with 22ecb9ba5c13b5a527c88784e63a07c01ce012e47b1f9eacabf87c056424e802 not found: ID does not exist" containerID="22ecb9ba5c13b5a527c88784e63a07c01ce012e47b1f9eacabf87c056424e802"
Jul 06 20:04:37.255998 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.255915    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ecb9ba5c13b5a527c88784e63a07c01ce012e47b1f9eacabf87c056424e802"} err="failed to get container status \"22ecb9ba5c13b5a527c88784e63a07c01ce012e47b1f9eacabf87c056424e802\": rpc error: code = NotFound desc = could not find container \"22ecb9ba5c13b5a527c88784e63a07c01ce012e47b1f9eacabf87c056424e802\": container with ID starting with 22ecb9ba5c13b5a527c88784e63a07c01ce012e47b1f9eacabf87c056424e802 not found: ID does not exist"
Jul 06 20:04:37.271478 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.271447    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"]
Jul 06 20:04:37.272761 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.272740    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-84fa8f39-6bdb994474-qw57x"]
Jul 06 20:04:37.895716 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:37.895686    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9db4314-247f-4a6f-a1dd-3d970f254277" path="/var/lib/kubelet/pods/b9db4314-247f-4a6f-a1dd-3d970f254277/volumes"
Jul 06 20:04:42.750865 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:42.750831    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:04:44.419423 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:44.419395    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:04:45.094326 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:45.094294    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:04:47.799260 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:04:47.799226    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:02.046783 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:02.046753    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:17.052815 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:17.052768    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:17.232560 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:17.232530    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:23.928544 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:23.928439    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:25.061759 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:25.061724    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:30.818495 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:30.818450    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:37.162954 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:37.162911    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:37.528616 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:37.528587    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:37.789523 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:37.789430    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:37.948447 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:37.948415    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:38.785063 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:38.785031    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:38.894292 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:38.894265    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:39.763820 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:39.763785    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:40.094710 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:40.094617    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:41.342278 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:41.342238    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:47.877643 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:47.877589    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:48.042082 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:48.042048    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:48.571542 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:48.571501    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:54.688655 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:54.688610    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:05:55.013419 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:05:55.013374    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:05.373751 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:05.373716    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:05.694506 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:05.694393    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:26.683922 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:26.683875    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:32.713136 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:32.713094    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:32.835915 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:32.835868    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:33.061029 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:33.060995    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:33.430140 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:33.430051    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:33.638598 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:33.638570    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:36.202016 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.201971    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:36.287090 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.287055    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"]
Jul 06 20:06:36.287415 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.287403    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9db4314-247f-4a6f-a1dd-3d970f254277" containerName="maas-api"
Jul 06 20:06:36.287458 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.287416    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9db4314-247f-4a6f-a1dd-3d970f254277" containerName="maas-api"
Jul 06 20:06:36.287513 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.287469    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9db4314-247f-4a6f-a1dd-3d970f254277" containerName="maas-api"
Jul 06 20:06:36.290293 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.290278    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"
Jul 06 20:06:36.293576 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.293556    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-cmcxr\""
Jul 06 20:06:36.293699 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.293634    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-d30688-serving-cert\""
Jul 06 20:06:36.305312 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.305292    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"]
Jul 06 20:06:36.349776 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.349749    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt5ll\" (UniqueName: \"kubernetes.io/projected/3d05e856-d334-49bf-9441-42c286781bd2-kube-api-access-qt5ll\") pod \"maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2\" (UID: \"3d05e856-d334-49bf-9441-42c286781bd2\") " pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"
Jul 06 20:06:36.349867 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.349808    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d05e856-d334-49bf-9441-42c286781bd2-maas-api-tls\") pod \"maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2\" (UID: \"3d05e856-d334-49bf-9441-42c286781bd2\") " pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"
Jul 06 20:06:36.349867 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.349832    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d05e856-d334-49bf-9441-42c286781bd2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2\" (UID: \"3d05e856-d334-49bf-9441-42c286781bd2\") " pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"
Jul 06 20:06:36.353625 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.353606    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:36.451293 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.451269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt5ll\" (UniqueName: \"kubernetes.io/projected/3d05e856-d334-49bf-9441-42c286781bd2-kube-api-access-qt5ll\") pod \"maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2\" (UID: \"3d05e856-d334-49bf-9441-42c286781bd2\") " pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"
Jul 06 20:06:36.451405 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.451313    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d05e856-d334-49bf-9441-42c286781bd2-maas-api-tls\") pod \"maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2\" (UID: \"3d05e856-d334-49bf-9441-42c286781bd2\") " pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"
Jul 06 20:06:36.451405 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.451332    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d05e856-d334-49bf-9441-42c286781bd2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2\" (UID: \"3d05e856-d334-49bf-9441-42c286781bd2\") " pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"
Jul 06 20:06:36.451954 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.451928    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d05e856-d334-49bf-9441-42c286781bd2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2\" (UID: \"3d05e856-d334-49bf-9441-42c286781bd2\") " pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"
Jul 06 20:06:36.453837 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.453785    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d05e856-d334-49bf-9441-42c286781bd2-maas-api-tls\") pod \"maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2\" (UID: \"3d05e856-d334-49bf-9441-42c286781bd2\") " pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"
Jul 06 20:06:36.459067 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.459030    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:36.462556 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.462533    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt5ll\" (UniqueName: \"kubernetes.io/projected/3d05e856-d334-49bf-9441-42c286781bd2-kube-api-access-qt5ll\") pod \"maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2\" (UID: \"3d05e856-d334-49bf-9441-42c286781bd2\") " pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"
Jul 06 20:06:36.581040 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.581014    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:36.599950 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.599928    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"
Jul 06 20:06:36.739475 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.739454    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"]
Jul 06 20:06:36.741422 ip-10-0-128-200 kubenswrapper[2562]: W0706 20:06:36.741398    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d05e856_d334_49bf_9441_42c286781bd2.slice/crio-d7950516e7297870a6c74603084f10266a869292e44a70105be76535c5eda679 WatchSource:0}: Error finding container d7950516e7297870a6c74603084f10266a869292e44a70105be76535c5eda679: Status 404 returned error can't find the container with id d7950516e7297870a6c74603084f10266a869292e44a70105be76535c5eda679
Jul 06 20:06:36.742793 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.742778    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 20:06:36.806454 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:36.806429    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2" event={"ID":"3d05e856-d334-49bf-9441-42c286781bd2","Type":"ContainerStarted","Data":"d7950516e7297870a6c74603084f10266a869292e44a70105be76535c5eda679"}
Jul 06 20:06:37.005690 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:37.005659    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:37.132937 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:37.132902    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:37.812172 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:37.812139    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2" event={"ID":"3d05e856-d334-49bf-9441-42c286781bd2","Type":"ContainerStarted","Data":"566d234f876a311c05bb14dac15028ccf5ae64bb0989a1277694092cda5c1eab"}
Jul 06 20:06:37.812564 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:37.812208    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"
Jul 06 20:06:37.838437 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:37.838390    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2" podStartSLOduration=1.42826471 podStartE2EDuration="1.838376188s" totalImagesPullingTime="410.111478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:06:37.837540809 +0000 UTC m=+2046.378717485" watchObservedRunningTime="2026-07-06 20:06:37.838376188 +0000 UTC m=+2046.379552863"
Jul 06 20:06:40.695004 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:40.694961    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:41.034086 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:41.034047    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:41.394832 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:41.394748    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:41.624668 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:41.624639    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:43.822534 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:43.822507    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"
Jul 06 20:06:47.349642 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:47.349606    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:47.464196 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:47.464166    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"]
Jul 06 20:06:47.467594 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:47.467577    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"
Jul 06 20:06:47.474410 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:47.474393    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-f78f77-serving-cert\""
Jul 06 20:06:47.495023 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:47.494997    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"]
Jul 06 20:06:47.538616 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:47.538592    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8khr\" (UniqueName: \"kubernetes.io/projected/d646e090-5ffb-40ae-9464-01e3228c3186-kube-api-access-s8khr\") pod \"maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl\" (UID: \"d646e090-5ffb-40ae-9464-01e3228c3186\") " pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"
Jul 06 20:06:47.538736 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:47.538643    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d646e090-5ffb-40ae-9464-01e3228c3186-maas-api-tls\") pod \"maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl\" (UID: \"d646e090-5ffb-40ae-9464-01e3228c3186\") " pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"
Jul 06 20:06:47.538736 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:47.538706    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d646e090-5ffb-40ae-9464-01e3228c3186-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl\" (UID: \"d646e090-5ffb-40ae-9464-01e3228c3186\") " pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"
Jul 06 20:06:47.550385 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:47.550359    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:47.639218 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:47.639162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8khr\" (UniqueName: \"kubernetes.io/projected/d646e090-5ffb-40ae-9464-01e3228c3186-kube-api-access-s8khr\") pod \"maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl\" (UID: \"d646e090-5ffb-40ae-9464-01e3228c3186\") " pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"
Jul 06 20:06:47.639218 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:47.639199    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d646e090-5ffb-40ae-9464-01e3228c3186-maas-api-tls\") pod \"maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl\" (UID: \"d646e090-5ffb-40ae-9464-01e3228c3186\") " pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"
Jul 06 20:06:47.639337 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:47.639227    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d646e090-5ffb-40ae-9464-01e3228c3186-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl\" (UID: \"d646e090-5ffb-40ae-9464-01e3228c3186\") " pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"
Jul 06 20:06:47.639337 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:06:47.639309    2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-f78f77-serving-cert: secret "maas-api-e2e-inf-b-f78f77-serving-cert" not found
Jul 06 20:06:47.639407 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:06:47.639376    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d646e090-5ffb-40ae-9464-01e3228c3186-maas-api-tls podName:d646e090-5ffb-40ae-9464-01e3228c3186 nodeName:}" failed. No retries permitted until 2026-07-06 20:06:48.13936147 +0000 UTC m=+2056.680538123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d646e090-5ffb-40ae-9464-01e3228c3186-maas-api-tls") pod "maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl" (UID: "d646e090-5ffb-40ae-9464-01e3228c3186") : secret "maas-api-e2e-inf-b-f78f77-serving-cert" not found
Jul 06 20:06:47.639856 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:47.639839    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d646e090-5ffb-40ae-9464-01e3228c3186-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl\" (UID: \"d646e090-5ffb-40ae-9464-01e3228c3186\") " pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"
Jul 06 20:06:47.663264 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:47.663236    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8khr\" (UniqueName: \"kubernetes.io/projected/d646e090-5ffb-40ae-9464-01e3228c3186-kube-api-access-s8khr\") pod \"maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl\" (UID: \"d646e090-5ffb-40ae-9464-01e3228c3186\") " pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"
Jul 06 20:06:47.764643 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:47.764616    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:47.963168 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:47.963106    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:48.143842 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:48.143812    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d646e090-5ffb-40ae-9464-01e3228c3186-maas-api-tls\") pod \"maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl\" (UID: \"d646e090-5ffb-40ae-9464-01e3228c3186\") " pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"
Jul 06 20:06:48.146216 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:48.146191    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d646e090-5ffb-40ae-9464-01e3228c3186-maas-api-tls\") pod \"maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl\" (UID: \"d646e090-5ffb-40ae-9464-01e3228c3186\") " pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"
Jul 06 20:06:48.264984 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:48.264943    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:48.377669 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:48.377645    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"
Jul 06 20:06:48.480228 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:48.480200    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:48.777108 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:48.776948    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"]
Jul 06 20:06:48.779979 ip-10-0-128-200 kubenswrapper[2562]: W0706 20:06:48.779953    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd646e090_5ffb_40ae_9464_01e3228c3186.slice/crio-360d9478ad1cc016ed7240f2ab7e19e2eaeccf0939fe290c5e9a894fd39f6734 WatchSource:0}: Error finding container 360d9478ad1cc016ed7240f2ab7e19e2eaeccf0939fe290c5e9a894fd39f6734: Status 404 returned error can't find the container with id 360d9478ad1cc016ed7240f2ab7e19e2eaeccf0939fe290c5e9a894fd39f6734
Jul 06 20:06:48.863518 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:48.863469    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl" event={"ID":"d646e090-5ffb-40ae-9464-01e3228c3186","Type":"ContainerStarted","Data":"360d9478ad1cc016ed7240f2ab7e19e2eaeccf0939fe290c5e9a894fd39f6734"}
Jul 06 20:06:49.870410 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:49.870367    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl" event={"ID":"d646e090-5ffb-40ae-9464-01e3228c3186","Type":"ContainerStarted","Data":"0919452d305310c083623a07d75b2896fe8f792ce34d5efb50a85ec27a2b01ce"}
Jul 06 20:06:49.870934 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:49.870555    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"
Jul 06 20:06:49.918471 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:49.918422    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl" podStartSLOduration=2.56916058 podStartE2EDuration="2.918407633s" totalImagesPullingTime="349.247053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:06:49.917329984 +0000 UTC m=+2058.458506661" watchObservedRunningTime="2026-07-06 20:06:49.918407633 +0000 UTC m=+2058.459584309"
Jul 06 20:06:51.799338 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:51.799300    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:55.880686 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:55.880601    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"
Jul 06 20:06:57.959858 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:57.959820    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:58.170681 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:58.170647    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:58.767336 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:58.767307    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"]
Jul 06 20:06:58.770752 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:58.770736    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:58.775781 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:58.775761    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d30688\"/\"openshift-service-ca.crt\""
Jul 06 20:06:58.775898 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:58.775877    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d30688\"/\"default-dockercfg-zkscq\""
Jul 06 20:06:58.775951 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:58.775877    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d30688\"/\"kube-root-ca.crt\""
Jul 06 20:06:58.775990 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:58.775879    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d30688\"/\"test-model-d30688-kserve-self-signed-certs\""
Jul 06 20:06:58.794179 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:58.794156    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"]
Jul 06 20:06:58.821782 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:58.821750    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:06:58.924559 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:58.924536    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ded338b-12da-4cca-bb6c-250eb89cd90d-tls-certs\") pod \"test-model-d30688-kserve-67c57b4bff-qwx56\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") " pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:58.924677 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:58.924566    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-dshm\") pod \"test-model-d30688-kserve-67c57b4bff-qwx56\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") " pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:58.924677 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:58.924583    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-model-cache\") pod \"test-model-d30688-kserve-67c57b4bff-qwx56\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") " pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:58.924677 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:58.924598    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-tmp-dir\") pod \"test-model-d30688-kserve-67c57b4bff-qwx56\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") " pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:58.924839 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:58.924682    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-home\") pod \"test-model-d30688-kserve-67c57b4bff-qwx56\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") " pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:58.924839 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:58.924802    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54xf2\" (UniqueName: \"kubernetes.io/projected/1ded338b-12da-4cca-bb6c-250eb89cd90d-kube-api-access-54xf2\") pod \"test-model-d30688-kserve-67c57b4bff-qwx56\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") " pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:59.025444 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:59.025384    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ded338b-12da-4cca-bb6c-250eb89cd90d-tls-certs\") pod \"test-model-d30688-kserve-67c57b4bff-qwx56\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") " pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:59.025444 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:59.025411    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-dshm\") pod \"test-model-d30688-kserve-67c57b4bff-qwx56\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") " pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:59.025444 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:59.025428    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-model-cache\") pod \"test-model-d30688-kserve-67c57b4bff-qwx56\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") " pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:59.025919 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:59.025446    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-tmp-dir\") pod \"test-model-d30688-kserve-67c57b4bff-qwx56\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") " pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:59.025919 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:59.025508    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-home\") pod \"test-model-d30688-kserve-67c57b4bff-qwx56\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") " pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:59.025919 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:59.025585    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54xf2\" (UniqueName: \"kubernetes.io/projected/1ded338b-12da-4cca-bb6c-250eb89cd90d-kube-api-access-54xf2\") pod \"test-model-d30688-kserve-67c57b4bff-qwx56\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") " pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:59.025919 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:59.025905    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-model-cache\") pod \"test-model-d30688-kserve-67c57b4bff-qwx56\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") " pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:59.026139 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:59.025936    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-tmp-dir\") pod \"test-model-d30688-kserve-67c57b4bff-qwx56\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") " pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:59.026139 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:59.025954    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-home\") pod \"test-model-d30688-kserve-67c57b4bff-qwx56\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") " pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:59.027784 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:59.027763    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-dshm\") pod \"test-model-d30688-kserve-67c57b4bff-qwx56\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") " pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:59.027890 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:59.027874    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ded338b-12da-4cca-bb6c-250eb89cd90d-tls-certs\") pod \"test-model-d30688-kserve-67c57b4bff-qwx56\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") " pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:59.034406 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:59.034386    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54xf2\" (UniqueName: \"kubernetes.io/projected/1ded338b-12da-4cca-bb6c-250eb89cd90d-kube-api-access-54xf2\") pod \"test-model-d30688-kserve-67c57b4bff-qwx56\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") " pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:59.080511 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:59.080472    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:06:59.214085 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:59.214054    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"]
Jul 06 20:06:59.217878 ip-10-0-128-200 kubenswrapper[2562]: W0706 20:06:59.217853    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ded338b_12da_4cca_bb6c_250eb89cd90d.slice/crio-094d3e3575d684f5f6d0380a1c4eb94ad68819be6d338fd9e9be7bc60a5d1c9b WatchSource:0}: Error finding container 094d3e3575d684f5f6d0380a1c4eb94ad68819be6d338fd9e9be7bc60a5d1c9b: Status 404 returned error can't find the container with id 094d3e3575d684f5f6d0380a1c4eb94ad68819be6d338fd9e9be7bc60a5d1c9b
Jul 06 20:06:59.919591 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:59.919553    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56" event={"ID":"1ded338b-12da-4cca-bb6c-250eb89cd90d","Type":"ContainerStarted","Data":"44a011493e98e9e95dcbd2de650b4a9a0d2d2c71a29c67a2f4b0126ae8471c98"}
Jul 06 20:06:59.919591 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:59.919598    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56" event={"ID":"1ded338b-12da-4cca-bb6c-250eb89cd90d","Type":"ContainerStarted","Data":"094d3e3575d684f5f6d0380a1c4eb94ad68819be6d338fd9e9be7bc60a5d1c9b"}
Jul 06 20:06:59.956465 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:06:59.956413    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56" podStartSLOduration=1.9563980170000002 podStartE2EDuration="1.956398017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:06:59.954887398 +0000 UTC m=+2068.496064066" watchObservedRunningTime="2026-07-06 20:06:59.956398017 +0000 UTC m=+2068.497574693"
Jul 06 20:07:02.902918 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:02.902889    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:07:03.111940 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:03.111908    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:07:09.080757 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:09.080727    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:07:09.081295 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:09.080772    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:07:09.093545 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:09.093505    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:07:09.589227 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:09.589197    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:07:09.788282 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:09.788249    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:07:09.973311 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:09.973249    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:07:32.548732 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:32.548690    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:07:50.392032 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:50.391999    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"]
Jul 06 20:07:50.447466 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:50.447441    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:07:51.462701 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:51.462661    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:07:52.006227 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:52.006196    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:07:52.118201 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:52.118170    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:07:52.262302 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:52.262234    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:07:52.362286 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:52.362257    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:07:52.423651 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:52.423619    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"]
Jul 06 20:07:52.423881 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:52.423859    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2" podUID="3d05e856-d334-49bf-9441-42c286781bd2" containerName="maas-api" containerID="cri-o://566d234f876a311c05bb14dac15028ccf5ae64bb0989a1277694092cda5c1eab" gracePeriod=30
Jul 06 20:07:52.473975 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:52.473951    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:07:52.665582 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:52.665555    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"
Jul 06 20:07:52.727183 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:52.727160    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qt5ll\" (UniqueName: \"kubernetes.io/projected/3d05e856-d334-49bf-9441-42c286781bd2-kube-api-access-qt5ll\") pod \"3d05e856-d334-49bf-9441-42c286781bd2\" (UID: \"3d05e856-d334-49bf-9441-42c286781bd2\") "
Jul 06 20:07:52.727308 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:52.727221    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d05e856-d334-49bf-9441-42c286781bd2-openshift-service-ca\") pod \"3d05e856-d334-49bf-9441-42c286781bd2\" (UID: \"3d05e856-d334-49bf-9441-42c286781bd2\") "
Jul 06 20:07:52.727308 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:52.727251    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d05e856-d334-49bf-9441-42c286781bd2-maas-api-tls\") pod \"3d05e856-d334-49bf-9441-42c286781bd2\" (UID: \"3d05e856-d334-49bf-9441-42c286781bd2\") "
Jul 06 20:07:52.727576 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:52.727553    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d05e856-d334-49bf-9441-42c286781bd2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3d05e856-d334-49bf-9441-42c286781bd2" (UID: "3d05e856-d334-49bf-9441-42c286781bd2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:07:52.729102 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:52.729079    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d05e856-d334-49bf-9441-42c286781bd2-kube-api-access-qt5ll" (OuterVolumeSpecName: "kube-api-access-qt5ll") pod "3d05e856-d334-49bf-9441-42c286781bd2" (UID: "3d05e856-d334-49bf-9441-42c286781bd2"). InnerVolumeSpecName "kube-api-access-qt5ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:07:52.729204 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:52.729138    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d05e856-d334-49bf-9441-42c286781bd2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3d05e856-d334-49bf-9441-42c286781bd2" (UID: "3d05e856-d334-49bf-9441-42c286781bd2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:07:52.828050 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:52.827995    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt5ll\" (UniqueName: \"kubernetes.io/projected/3d05e856-d334-49bf-9441-42c286781bd2-kube-api-access-qt5ll\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:07:52.828050 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:52.828017    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d05e856-d334-49bf-9441-42c286781bd2-openshift-service-ca\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:07:52.828050 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:52.828027    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d05e856-d334-49bf-9441-42c286781bd2-maas-api-tls\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:07:53.169130 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:53.169045    2562 generic.go:356] "Generic (PLEG): container finished" podID="3d05e856-d334-49bf-9441-42c286781bd2" containerID="566d234f876a311c05bb14dac15028ccf5ae64bb0989a1277694092cda5c1eab" exitCode=0
Jul 06 20:07:53.169130 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:53.169087    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"
Jul 06 20:07:53.169346 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:53.169162    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2" event={"ID":"3d05e856-d334-49bf-9441-42c286781bd2","Type":"ContainerDied","Data":"566d234f876a311c05bb14dac15028ccf5ae64bb0989a1277694092cda5c1eab"}
Jul 06 20:07:53.169346 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:53.169202    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2" event={"ID":"3d05e856-d334-49bf-9441-42c286781bd2","Type":"ContainerDied","Data":"d7950516e7297870a6c74603084f10266a869292e44a70105be76535c5eda679"}
Jul 06 20:07:53.169346 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:53.169225    2562 scope.go:117] "RemoveContainer" containerID="566d234f876a311c05bb14dac15028ccf5ae64bb0989a1277694092cda5c1eab"
Jul 06 20:07:53.178802 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:53.178763    2562 scope.go:117] "RemoveContainer" containerID="566d234f876a311c05bb14dac15028ccf5ae64bb0989a1277694092cda5c1eab"
Jul 06 20:07:53.179140 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:07:53.179105    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566d234f876a311c05bb14dac15028ccf5ae64bb0989a1277694092cda5c1eab\": container with ID starting with 566d234f876a311c05bb14dac15028ccf5ae64bb0989a1277694092cda5c1eab not found: ID does not exist" containerID="566d234f876a311c05bb14dac15028ccf5ae64bb0989a1277694092cda5c1eab"
Jul 06 20:07:53.179230 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:53.179140    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566d234f876a311c05bb14dac15028ccf5ae64bb0989a1277694092cda5c1eab"} err="failed to get container status \"566d234f876a311c05bb14dac15028ccf5ae64bb0989a1277694092cda5c1eab\": rpc error: code = NotFound desc = could not find container \"566d234f876a311c05bb14dac15028ccf5ae64bb0989a1277694092cda5c1eab\": container with ID starting with 566d234f876a311c05bb14dac15028ccf5ae64bb0989a1277694092cda5c1eab not found: ID does not exist"
Jul 06 20:07:53.187598 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:07:53.187576    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d05e856_d334_49bf_9441_42c286781bd2.slice/crio-d7950516e7297870a6c74603084f10266a869292e44a70105be76535c5eda679\": RecentStats: unable to find data in memory cache]"
Jul 06 20:07:53.187687 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:07:53.187635    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d05e856_d334_49bf_9441_42c286781bd2.slice\": RecentStats: unable to find data in memory cache]"
Jul 06 20:07:53.199177 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:53.199158    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"]
Jul 06 20:07:53.204008 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:53.203988    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d30688-67cdd9b54d-c6nr2"]
Jul 06 20:07:53.895358 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:53.895324    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d05e856-d334-49bf-9441-42c286781bd2" path="/var/lib/kubelet/pods/3d05e856-d334-49bf-9441-42c286781bd2/volumes"
Jul 06 20:07:57.080059 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:07:57.080014    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:05.411844 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.411800    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56" podUID="1ded338b-12da-4cca-bb6c-250eb89cd90d" containerName="main" containerID="cri-o://44a011493e98e9e95dcbd2de650b4a9a0d2d2c71a29c67a2f4b0126ae8471c98" gracePeriod=45
Jul 06 20:08:05.668464 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.668410    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:08:05.727190 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.727164    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ded338b-12da-4cca-bb6c-250eb89cd90d-tls-certs\") pod \"1ded338b-12da-4cca-bb6c-250eb89cd90d\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") "
Jul 06 20:08:05.727311 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.727205    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-home\") pod \"1ded338b-12da-4cca-bb6c-250eb89cd90d\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") "
Jul 06 20:08:05.727311 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.727268    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-dshm\") pod \"1ded338b-12da-4cca-bb6c-250eb89cd90d\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") "
Jul 06 20:08:05.727427 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.727330    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-tmp-dir\") pod \"1ded338b-12da-4cca-bb6c-250eb89cd90d\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") "
Jul 06 20:08:05.727427 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.727358    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-model-cache\") pod \"1ded338b-12da-4cca-bb6c-250eb89cd90d\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") "
Jul 06 20:08:05.727427 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.727397    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54xf2\" (UniqueName: \"kubernetes.io/projected/1ded338b-12da-4cca-bb6c-250eb89cd90d-kube-api-access-54xf2\") pod \"1ded338b-12da-4cca-bb6c-250eb89cd90d\" (UID: \"1ded338b-12da-4cca-bb6c-250eb89cd90d\") "
Jul 06 20:08:05.727606 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.727565    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-home" (OuterVolumeSpecName: "home") pod "1ded338b-12da-4cca-bb6c-250eb89cd90d" (UID: "1ded338b-12da-4cca-bb6c-250eb89cd90d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:08:05.727666 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.727644    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ded338b-12da-4cca-bb6c-250eb89cd90d" (UID: "1ded338b-12da-4cca-bb6c-250eb89cd90d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:08:05.727712 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.727655    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ded338b-12da-4cca-bb6c-250eb89cd90d" (UID: "1ded338b-12da-4cca-bb6c-250eb89cd90d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:08:05.727774 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.727756    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-tmp-dir\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:08:05.727817 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.727780    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-model-cache\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:08:05.727817 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.727792    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-home\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:08:05.729390 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.729360    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-dshm" (OuterVolumeSpecName: "dshm") pod "1ded338b-12da-4cca-bb6c-250eb89cd90d" (UID: "1ded338b-12da-4cca-bb6c-250eb89cd90d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:08:05.729519 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.729387    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ded338b-12da-4cca-bb6c-250eb89cd90d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ded338b-12da-4cca-bb6c-250eb89cd90d" (UID: "1ded338b-12da-4cca-bb6c-250eb89cd90d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:08:05.729519 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.729422    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ded338b-12da-4cca-bb6c-250eb89cd90d-kube-api-access-54xf2" (OuterVolumeSpecName: "kube-api-access-54xf2") pod "1ded338b-12da-4cca-bb6c-250eb89cd90d" (UID: "1ded338b-12da-4cca-bb6c-250eb89cd90d"). InnerVolumeSpecName "kube-api-access-54xf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:08:05.829129 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.829106    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54xf2\" (UniqueName: \"kubernetes.io/projected/1ded338b-12da-4cca-bb6c-250eb89cd90d-kube-api-access-54xf2\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:08:05.829129 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.829127    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ded338b-12da-4cca-bb6c-250eb89cd90d-tls-certs\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:08:05.829249 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:05.829137    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ded338b-12da-4cca-bb6c-250eb89cd90d-dshm\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:08:06.231404 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:06.231369    2562 generic.go:356] "Generic (PLEG): container finished" podID="1ded338b-12da-4cca-bb6c-250eb89cd90d" containerID="44a011493e98e9e95dcbd2de650b4a9a0d2d2c71a29c67a2f4b0126ae8471c98" exitCode=0
Jul 06 20:08:06.231614 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:06.231413    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"
Jul 06 20:08:06.231614 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:06.231426    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56" event={"ID":"1ded338b-12da-4cca-bb6c-250eb89cd90d","Type":"ContainerDied","Data":"44a011493e98e9e95dcbd2de650b4a9a0d2d2c71a29c67a2f4b0126ae8471c98"}
Jul 06 20:08:06.231614 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:06.231464    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56" event={"ID":"1ded338b-12da-4cca-bb6c-250eb89cd90d","Type":"ContainerDied","Data":"094d3e3575d684f5f6d0380a1c4eb94ad68819be6d338fd9e9be7bc60a5d1c9b"}
Jul 06 20:08:06.231614 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:06.231519    2562 scope.go:117] "RemoveContainer" containerID="44a011493e98e9e95dcbd2de650b4a9a0d2d2c71a29c67a2f4b0126ae8471c98"
Jul 06 20:08:06.239458 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:06.239441    2562 scope.go:117] "RemoveContainer" containerID="44a011493e98e9e95dcbd2de650b4a9a0d2d2c71a29c67a2f4b0126ae8471c98"
Jul 06 20:08:06.239713 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:08:06.239693    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a011493e98e9e95dcbd2de650b4a9a0d2d2c71a29c67a2f4b0126ae8471c98\": container with ID starting with 44a011493e98e9e95dcbd2de650b4a9a0d2d2c71a29c67a2f4b0126ae8471c98 not found: ID does not exist" containerID="44a011493e98e9e95dcbd2de650b4a9a0d2d2c71a29c67a2f4b0126ae8471c98"
Jul 06 20:08:06.239801 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:06.239720    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a011493e98e9e95dcbd2de650b4a9a0d2d2c71a29c67a2f4b0126ae8471c98"} err="failed to get container status \"44a011493e98e9e95dcbd2de650b4a9a0d2d2c71a29c67a2f4b0126ae8471c98\": rpc error: code = NotFound desc = could not find container \"44a011493e98e9e95dcbd2de650b4a9a0d2d2c71a29c67a2f4b0126ae8471c98\": container with ID starting with 44a011493e98e9e95dcbd2de650b4a9a0d2d2c71a29c67a2f4b0126ae8471c98 not found: ID does not exist"
Jul 06 20:08:06.250334 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:06.250313    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"]
Jul 06 20:08:06.254665 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:06.254647    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d30688/test-model-d30688-kserve-67c57b4bff-qwx56"]
Jul 06 20:08:07.894783 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:07.894752    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ded338b-12da-4cca-bb6c-250eb89cd90d" path="/var/lib/kubelet/pods/1ded338b-12da-4cca-bb6c-250eb89cd90d/volumes"
Jul 06 20:08:31.304051 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:31.303966    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:32.858242 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:32.858207    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:34.359309 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:34.359264    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:34.653957 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:34.653874    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:34.892172 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:34.892138    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:35.172850 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:35.172823    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:35.529232 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:35.529204    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:35.640625 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:35.640595    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:35.749989 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:35.749958    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:35.836228 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:35.836163    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"]
Jul 06 20:08:35.836647 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:35.836617    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl" podUID="d646e090-5ffb-40ae-9464-01e3228c3186" containerName="maas-api" containerID="cri-o://0919452d305310c083623a07d75b2896fe8f792ce34d5efb50a85ec27a2b01ce" gracePeriod=30
Jul 06 20:08:35.876340 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:35.876313    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl" podUID="d646e090-5ffb-40ae-9464-01e3228c3186" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.40:8443/health\": dial tcp 10.132.0.40:8443: connect: connection refused"
Jul 06 20:08:35.890536 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:35.890511    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:36.088229 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:36.088181    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"
Jul 06 20:08:36.157068 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:36.157043    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d646e090-5ffb-40ae-9464-01e3228c3186-maas-api-tls\") pod \"d646e090-5ffb-40ae-9464-01e3228c3186\" (UID: \"d646e090-5ffb-40ae-9464-01e3228c3186\") "
Jul 06 20:08:36.157198 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:36.157072    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d646e090-5ffb-40ae-9464-01e3228c3186-openshift-service-ca\") pod \"d646e090-5ffb-40ae-9464-01e3228c3186\" (UID: \"d646e090-5ffb-40ae-9464-01e3228c3186\") "
Jul 06 20:08:36.157198 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:36.157101    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8khr\" (UniqueName: \"kubernetes.io/projected/d646e090-5ffb-40ae-9464-01e3228c3186-kube-api-access-s8khr\") pod \"d646e090-5ffb-40ae-9464-01e3228c3186\" (UID: \"d646e090-5ffb-40ae-9464-01e3228c3186\") "
Jul 06 20:08:36.157427 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:36.157406    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d646e090-5ffb-40ae-9464-01e3228c3186-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d646e090-5ffb-40ae-9464-01e3228c3186" (UID: "d646e090-5ffb-40ae-9464-01e3228c3186"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:08:36.159036 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:36.159016    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d646e090-5ffb-40ae-9464-01e3228c3186-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d646e090-5ffb-40ae-9464-01e3228c3186" (UID: "d646e090-5ffb-40ae-9464-01e3228c3186"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:08:36.159125 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:36.159079    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d646e090-5ffb-40ae-9464-01e3228c3186-kube-api-access-s8khr" (OuterVolumeSpecName: "kube-api-access-s8khr") pod "d646e090-5ffb-40ae-9464-01e3228c3186" (UID: "d646e090-5ffb-40ae-9464-01e3228c3186"). InnerVolumeSpecName "kube-api-access-s8khr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:08:36.257774 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:36.257750    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d646e090-5ffb-40ae-9464-01e3228c3186-maas-api-tls\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:08:36.257774 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:36.257773    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d646e090-5ffb-40ae-9464-01e3228c3186-openshift-service-ca\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:08:36.257953 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:36.257783    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8khr\" (UniqueName: \"kubernetes.io/projected/d646e090-5ffb-40ae-9464-01e3228c3186-kube-api-access-s8khr\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:08:36.377460 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:36.377394    2562 generic.go:356] "Generic (PLEG): container finished" podID="d646e090-5ffb-40ae-9464-01e3228c3186" containerID="0919452d305310c083623a07d75b2896fe8f792ce34d5efb50a85ec27a2b01ce" exitCode=0
Jul 06 20:08:36.377591 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:36.377461    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"
Jul 06 20:08:36.377591 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:36.377496    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl" event={"ID":"d646e090-5ffb-40ae-9464-01e3228c3186","Type":"ContainerDied","Data":"0919452d305310c083623a07d75b2896fe8f792ce34d5efb50a85ec27a2b01ce"}
Jul 06 20:08:36.377591 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:36.377574    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl" event={"ID":"d646e090-5ffb-40ae-9464-01e3228c3186","Type":"ContainerDied","Data":"360d9478ad1cc016ed7240f2ab7e19e2eaeccf0939fe290c5e9a894fd39f6734"}
Jul 06 20:08:36.377704 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:36.377597    2562 scope.go:117] "RemoveContainer" containerID="0919452d305310c083623a07d75b2896fe8f792ce34d5efb50a85ec27a2b01ce"
Jul 06 20:08:36.386707 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:36.386688    2562 scope.go:117] "RemoveContainer" containerID="0919452d305310c083623a07d75b2896fe8f792ce34d5efb50a85ec27a2b01ce"
Jul 06 20:08:36.386987 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:08:36.386961    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0919452d305310c083623a07d75b2896fe8f792ce34d5efb50a85ec27a2b01ce\": container with ID starting with 0919452d305310c083623a07d75b2896fe8f792ce34d5efb50a85ec27a2b01ce not found: ID does not exist" containerID="0919452d305310c083623a07d75b2896fe8f792ce34d5efb50a85ec27a2b01ce"
Jul 06 20:08:36.387056 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:36.386989    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0919452d305310c083623a07d75b2896fe8f792ce34d5efb50a85ec27a2b01ce"} err="failed to get container status \"0919452d305310c083623a07d75b2896fe8f792ce34d5efb50a85ec27a2b01ce\": rpc error: code = NotFound desc = could not find container \"0919452d305310c083623a07d75b2896fe8f792ce34d5efb50a85ec27a2b01ce\": container with ID starting with 0919452d305310c083623a07d75b2896fe8f792ce34d5efb50a85ec27a2b01ce not found: ID does not exist"
Jul 06 20:08:36.412723 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:36.412698    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"]
Jul 06 20:08:36.417388 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:36.417370    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f78f77-667544fcdf-bwbkl"]
Jul 06 20:08:37.894834 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:37.894798    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d646e090-5ffb-40ae-9464-01e3228c3186" path="/var/lib/kubelet/pods/d646e090-5ffb-40ae-9464-01e3228c3186/volumes"
Jul 06 20:08:42.879859 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:42.879821    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:42.979043 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:42.979008    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:43.200071 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:43.199994    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:43.437865 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:43.437818    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:43.803291 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:43.803254    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:44.187271 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:44.187196    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:44.382377 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:44.382341    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:46.947667 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:46.947634    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:47.074581 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.074550    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:47.202046 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.201976    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"]
Jul 06 20:08:47.202321 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.202309    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d646e090-5ffb-40ae-9464-01e3228c3186" containerName="maas-api"
Jul 06 20:08:47.202363 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.202323    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d646e090-5ffb-40ae-9464-01e3228c3186" containerName="maas-api"
Jul 06 20:08:47.202363 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.202333    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ded338b-12da-4cca-bb6c-250eb89cd90d" containerName="main"
Jul 06 20:08:47.202363 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.202339    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ded338b-12da-4cca-bb6c-250eb89cd90d" containerName="main"
Jul 06 20:08:47.202363 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.202357    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d05e856-d334-49bf-9441-42c286781bd2" containerName="maas-api"
Jul 06 20:08:47.202513 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.202363    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d05e856-d334-49bf-9441-42c286781bd2" containerName="maas-api"
Jul 06 20:08:47.202513 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.202422    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ded338b-12da-4cca-bb6c-250eb89cd90d" containerName="main"
Jul 06 20:08:47.202513 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.202432    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d05e856-d334-49bf-9441-42c286781bd2" containerName="maas-api"
Jul 06 20:08:47.202513 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.202438    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d646e090-5ffb-40ae-9464-01e3228c3186" containerName="maas-api"
Jul 06 20:08:47.205479 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.205462    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"
Jul 06 20:08:47.211456 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.211435    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-cmcxr\""
Jul 06 20:08:47.211615 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.211549    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-2131ed-serving-cert\""
Jul 06 20:08:47.229845 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.229823    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"]
Jul 06 20:08:47.337770 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.337742    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17545ef-b734-45e3-a0a3-d17b0546f4f3-openshift-service-ca\") pod \"maas-api-e2e-api-a-2131ed-84456855b6-f9q2z\" (UID: \"f17545ef-b734-45e3-a0a3-d17b0546f4f3\") " pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"
Jul 06 20:08:47.337918 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.337775    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plk5g\" (UniqueName: \"kubernetes.io/projected/f17545ef-b734-45e3-a0a3-d17b0546f4f3-kube-api-access-plk5g\") pod \"maas-api-e2e-api-a-2131ed-84456855b6-f9q2z\" (UID: \"f17545ef-b734-45e3-a0a3-d17b0546f4f3\") " pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"
Jul 06 20:08:47.337918 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.337812    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f17545ef-b734-45e3-a0a3-d17b0546f4f3-maas-api-tls\") pod \"maas-api-e2e-api-a-2131ed-84456855b6-f9q2z\" (UID: \"f17545ef-b734-45e3-a0a3-d17b0546f4f3\") " pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"
Jul 06 20:08:47.360160 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.360136    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:47.439106 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.439081    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17545ef-b734-45e3-a0a3-d17b0546f4f3-openshift-service-ca\") pod \"maas-api-e2e-api-a-2131ed-84456855b6-f9q2z\" (UID: \"f17545ef-b734-45e3-a0a3-d17b0546f4f3\") " pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"
Jul 06 20:08:47.439198 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.439113    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plk5g\" (UniqueName: \"kubernetes.io/projected/f17545ef-b734-45e3-a0a3-d17b0546f4f3-kube-api-access-plk5g\") pod \"maas-api-e2e-api-a-2131ed-84456855b6-f9q2z\" (UID: \"f17545ef-b734-45e3-a0a3-d17b0546f4f3\") " pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"
Jul 06 20:08:47.439259 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.439233    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f17545ef-b734-45e3-a0a3-d17b0546f4f3-maas-api-tls\") pod \"maas-api-e2e-api-a-2131ed-84456855b6-f9q2z\" (UID: \"f17545ef-b734-45e3-a0a3-d17b0546f4f3\") " pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"
Jul 06 20:08:47.439351 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:08:47.439337    2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-2131ed-serving-cert: secret "maas-api-e2e-api-a-2131ed-serving-cert" not found
Jul 06 20:08:47.439415 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:08:47.439406    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f17545ef-b734-45e3-a0a3-d17b0546f4f3-maas-api-tls podName:f17545ef-b734-45e3-a0a3-d17b0546f4f3 nodeName:}" failed. No retries permitted until 2026-07-06 20:08:47.939389391 +0000 UTC m=+2176.480566044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f17545ef-b734-45e3-a0a3-d17b0546f4f3-maas-api-tls") pod "maas-api-e2e-api-a-2131ed-84456855b6-f9q2z" (UID: "f17545ef-b734-45e3-a0a3-d17b0546f4f3") : secret "maas-api-e2e-api-a-2131ed-serving-cert" not found
Jul 06 20:08:47.439690 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.439674    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17545ef-b734-45e3-a0a3-d17b0546f4f3-openshift-service-ca\") pod \"maas-api-e2e-api-a-2131ed-84456855b6-f9q2z\" (UID: \"f17545ef-b734-45e3-a0a3-d17b0546f4f3\") " pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"
Jul 06 20:08:47.499710 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.499657    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plk5g\" (UniqueName: \"kubernetes.io/projected/f17545ef-b734-45e3-a0a3-d17b0546f4f3-kube-api-access-plk5g\") pod \"maas-api-e2e-api-a-2131ed-84456855b6-f9q2z\" (UID: \"f17545ef-b734-45e3-a0a3-d17b0546f4f3\") " pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"
Jul 06 20:08:47.710506 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.710439    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:47.941633 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.941609    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f17545ef-b734-45e3-a0a3-d17b0546f4f3-maas-api-tls\") pod \"maas-api-e2e-api-a-2131ed-84456855b6-f9q2z\" (UID: \"f17545ef-b734-45e3-a0a3-d17b0546f4f3\") " pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"
Jul 06 20:08:47.943837 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:47.943817    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f17545ef-b734-45e3-a0a3-d17b0546f4f3-maas-api-tls\") pod \"maas-api-e2e-api-a-2131ed-84456855b6-f9q2z\" (UID: \"f17545ef-b734-45e3-a0a3-d17b0546f4f3\") " pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"
Jul 06 20:08:48.116344 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:48.116290    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"
Jul 06 20:08:48.259910 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:48.259886    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"]
Jul 06 20:08:48.291294 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:48.291263    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:48.441571 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:48.441528    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z" event={"ID":"f17545ef-b734-45e3-a0a3-d17b0546f4f3","Type":"ContainerStarted","Data":"691ca271f3cbe706fba1613cd695c1d96c3572090c498c1ba5e22a9e62508cdc"}
Jul 06 20:08:48.463946 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:48.463871    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:49.446878 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:49.446842    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z" event={"ID":"f17545ef-b734-45e3-a0a3-d17b0546f4f3","Type":"ContainerStarted","Data":"83a7d139bd3c213af3d474aad379721792adf314b069e58683759bf8fc6f0d1b"}
Jul 06 20:08:49.447250 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:49.446919    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"
Jul 06 20:08:49.470055 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:49.470008    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z" podStartSLOduration=1.9525461910000002 podStartE2EDuration="2.46999469s" totalImagesPullingTime="517.448499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:08:49.4685108 +0000 UTC m=+2178.009687471" watchObservedRunningTime="2026-07-06 20:08:49.46999469 +0000 UTC m=+2178.011171367"
Jul 06 20:08:51.438169 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:51.438131    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:51.866647 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:51.866614    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:52.182930 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:52.182855    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:52.410522 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:52.410478    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:55.147476 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:55.147443    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:55.169759 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:55.169729    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"]
Jul 06 20:08:55.173230 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:55.173211    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"
Jul 06 20:08:55.176123 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:55.176102    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-78a644-serving-cert\""
Jul 06 20:08:55.184749 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:55.184724    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"]
Jul 06 20:08:55.205985 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:55.205957    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48837016-f3b3-4b0c-82c7-60936438fa82-openshift-service-ca\") pod \"maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk\" (UID: \"48837016-f3b3-4b0c-82c7-60936438fa82\") " pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"
Jul 06 20:08:55.206081 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:55.206006    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv4c8\" (UniqueName: \"kubernetes.io/projected/48837016-f3b3-4b0c-82c7-60936438fa82-kube-api-access-fv4c8\") pod \"maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk\" (UID: \"48837016-f3b3-4b0c-82c7-60936438fa82\") " pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"
Jul 06 20:08:55.206081 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:55.206073    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48837016-f3b3-4b0c-82c7-60936438fa82-maas-api-tls\") pod \"maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk\" (UID: \"48837016-f3b3-4b0c-82c7-60936438fa82\") " pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"
Jul 06 20:08:55.307408 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:55.307385    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48837016-f3b3-4b0c-82c7-60936438fa82-openshift-service-ca\") pod \"maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk\" (UID: \"48837016-f3b3-4b0c-82c7-60936438fa82\") " pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"
Jul 06 20:08:55.307551 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:55.307418    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv4c8\" (UniqueName: \"kubernetes.io/projected/48837016-f3b3-4b0c-82c7-60936438fa82-kube-api-access-fv4c8\") pod \"maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk\" (UID: \"48837016-f3b3-4b0c-82c7-60936438fa82\") " pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"
Jul 06 20:08:55.307551 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:55.307449    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48837016-f3b3-4b0c-82c7-60936438fa82-maas-api-tls\") pod \"maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk\" (UID: \"48837016-f3b3-4b0c-82c7-60936438fa82\") " pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"
Jul 06 20:08:55.307634 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:08:55.307612    2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-78a644-serving-cert: secret "maas-api-e2e-api-b-78a644-serving-cert" not found
Jul 06 20:08:55.307703 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:08:55.307691    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48837016-f3b3-4b0c-82c7-60936438fa82-maas-api-tls podName:48837016-f3b3-4b0c-82c7-60936438fa82 nodeName:}" failed. No retries permitted until 2026-07-06 20:08:55.807669981 +0000 UTC m=+2184.348846640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/48837016-f3b3-4b0c-82c7-60936438fa82-maas-api-tls") pod "maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk" (UID: "48837016-f3b3-4b0c-82c7-60936438fa82") : secret "maas-api-e2e-api-b-78a644-serving-cert" not found
Jul 06 20:08:55.308139 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:55.308123    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48837016-f3b3-4b0c-82c7-60936438fa82-openshift-service-ca\") pod \"maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk\" (UID: \"48837016-f3b3-4b0c-82c7-60936438fa82\") " pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"
Jul 06 20:08:55.336669 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:55.336647    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:55.336990 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:55.336972    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv4c8\" (UniqueName: \"kubernetes.io/projected/48837016-f3b3-4b0c-82c7-60936438fa82-kube-api-access-fv4c8\") pod \"maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk\" (UID: \"48837016-f3b3-4b0c-82c7-60936438fa82\") " pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"
Jul 06 20:08:55.456436 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:55.456375    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"
Jul 06 20:08:55.583259 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:55.583219    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:55.812556 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:55.812528    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48837016-f3b3-4b0c-82c7-60936438fa82-maas-api-tls\") pod \"maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk\" (UID: \"48837016-f3b3-4b0c-82c7-60936438fa82\") " pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"
Jul 06 20:08:55.814941 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:55.814919    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48837016-f3b3-4b0c-82c7-60936438fa82-maas-api-tls\") pod \"maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk\" (UID: \"48837016-f3b3-4b0c-82c7-60936438fa82\") " pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"
Jul 06 20:08:56.061139 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:56.061104    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:56.085134 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:56.085078    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"
Jul 06 20:08:56.161688 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:56.161655    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:08:56.436889 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:56.436867    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"]
Jul 06 20:08:56.438916 ip-10-0-128-200 kubenswrapper[2562]: W0706 20:08:56.438892    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48837016_f3b3_4b0c_82c7_60936438fa82.slice/crio-5a2121b8a85179f29d6dc22a17df11b77782d11f43ac2961d16927b311fbed2f WatchSource:0}: Error finding container 5a2121b8a85179f29d6dc22a17df11b77782d11f43ac2961d16927b311fbed2f: Status 404 returned error can't find the container with id 5a2121b8a85179f29d6dc22a17df11b77782d11f43ac2961d16927b311fbed2f
Jul 06 20:08:56.478465 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:56.478436    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk" event={"ID":"48837016-f3b3-4b0c-82c7-60936438fa82","Type":"ContainerStarted","Data":"5a2121b8a85179f29d6dc22a17df11b77782d11f43ac2961d16927b311fbed2f"}
Jul 06 20:08:57.484784 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:57.484748    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk" event={"ID":"48837016-f3b3-4b0c-82c7-60936438fa82","Type":"ContainerStarted","Data":"f1e05d4b2d1f2ab1e7e75aee1b3ca7c2161ac63baf9ce3dd95013442d9583496"}
Jul 06 20:08:57.485184 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:57.484882    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"
Jul 06 20:08:57.514106 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:57.514058    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk" podStartSLOduration=2.114409564 podStartE2EDuration="2.514042522s" totalImagesPullingTime="399.632958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:08:57.511329616 +0000 UTC m=+2186.052506295" watchObservedRunningTime="2026-07-06 20:08:57.514042522 +0000 UTC m=+2186.055219197"
Jul 06 20:08:59.555709 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:08:59.555667    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:03.495680 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:03.495650    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"
Jul 06 20:09:07.757621 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:07.757586    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:08.212819 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:08.212738    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:08.473731 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:08.473650    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:08.677120 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:08.677087    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:08.718784 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:08.718756    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"]
Jul 06 20:09:08.719001 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:08.718978    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z" podUID="f17545ef-b734-45e3-a0a3-d17b0546f4f3" containerName="maas-api" containerID="cri-o://83a7d139bd3c213af3d474aad379721792adf314b069e58683759bf8fc6f0d1b" gracePeriod=30
Jul 06 20:09:08.824708 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:08.824681    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:08.966831 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:08.966811    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"
Jul 06 20:09:09.009764 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:09.009740    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plk5g\" (UniqueName: \"kubernetes.io/projected/f17545ef-b734-45e3-a0a3-d17b0546f4f3-kube-api-access-plk5g\") pod \"f17545ef-b734-45e3-a0a3-d17b0546f4f3\" (UID: \"f17545ef-b734-45e3-a0a3-d17b0546f4f3\") "
Jul 06 20:09:09.010081 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:09.009783    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17545ef-b734-45e3-a0a3-d17b0546f4f3-openshift-service-ca\") pod \"f17545ef-b734-45e3-a0a3-d17b0546f4f3\" (UID: \"f17545ef-b734-45e3-a0a3-d17b0546f4f3\") "
Jul 06 20:09:09.010081 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:09.009882    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f17545ef-b734-45e3-a0a3-d17b0546f4f3-maas-api-tls\") pod \"f17545ef-b734-45e3-a0a3-d17b0546f4f3\" (UID: \"f17545ef-b734-45e3-a0a3-d17b0546f4f3\") "
Jul 06 20:09:09.010214 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:09.010136    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f17545ef-b734-45e3-a0a3-d17b0546f4f3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f17545ef-b734-45e3-a0a3-d17b0546f4f3" (UID: "f17545ef-b734-45e3-a0a3-d17b0546f4f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:09:09.010272 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:09.010232    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17545ef-b734-45e3-a0a3-d17b0546f4f3-openshift-service-ca\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:09:09.011870 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:09.011847    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f17545ef-b734-45e3-a0a3-d17b0546f4f3-kube-api-access-plk5g" (OuterVolumeSpecName: "kube-api-access-plk5g") pod "f17545ef-b734-45e3-a0a3-d17b0546f4f3" (UID: "f17545ef-b734-45e3-a0a3-d17b0546f4f3"). InnerVolumeSpecName "kube-api-access-plk5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:09:09.011990 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:09.011908    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f17545ef-b734-45e3-a0a3-d17b0546f4f3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f17545ef-b734-45e3-a0a3-d17b0546f4f3" (UID: "f17545ef-b734-45e3-a0a3-d17b0546f4f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:09:09.111563 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:09.111537    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f17545ef-b734-45e3-a0a3-d17b0546f4f3-maas-api-tls\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:09:09.111563 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:09.111561    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plk5g\" (UniqueName: \"kubernetes.io/projected/f17545ef-b734-45e3-a0a3-d17b0546f4f3-kube-api-access-plk5g\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:09:09.540833 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:09.540800    2562 generic.go:356] "Generic (PLEG): container finished" podID="f17545ef-b734-45e3-a0a3-d17b0546f4f3" containerID="83a7d139bd3c213af3d474aad379721792adf314b069e58683759bf8fc6f0d1b" exitCode=0
Jul 06 20:09:09.540957 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:09.540844    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"
Jul 06 20:09:09.540957 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:09.540883    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z" event={"ID":"f17545ef-b734-45e3-a0a3-d17b0546f4f3","Type":"ContainerDied","Data":"83a7d139bd3c213af3d474aad379721792adf314b069e58683759bf8fc6f0d1b"}
Jul 06 20:09:09.540957 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:09.540934    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z" event={"ID":"f17545ef-b734-45e3-a0a3-d17b0546f4f3","Type":"ContainerDied","Data":"691ca271f3cbe706fba1613cd695c1d96c3572090c498c1ba5e22a9e62508cdc"}
Jul 06 20:09:09.541072 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:09.540957    2562 scope.go:117] "RemoveContainer" containerID="83a7d139bd3c213af3d474aad379721792adf314b069e58683759bf8fc6f0d1b"
Jul 06 20:09:09.549263 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:09.549248    2562 scope.go:117] "RemoveContainer" containerID="83a7d139bd3c213af3d474aad379721792adf314b069e58683759bf8fc6f0d1b"
Jul 06 20:09:09.549537 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:09:09.549519    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83a7d139bd3c213af3d474aad379721792adf314b069e58683759bf8fc6f0d1b\": container with ID starting with 83a7d139bd3c213af3d474aad379721792adf314b069e58683759bf8fc6f0d1b not found: ID does not exist" containerID="83a7d139bd3c213af3d474aad379721792adf314b069e58683759bf8fc6f0d1b"
Jul 06 20:09:09.549591 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:09.549546    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a7d139bd3c213af3d474aad379721792adf314b069e58683759bf8fc6f0d1b"} err="failed to get container status \"83a7d139bd3c213af3d474aad379721792adf314b069e58683759bf8fc6f0d1b\": rpc error: code = NotFound desc = could not find container \"83a7d139bd3c213af3d474aad379721792adf314b069e58683759bf8fc6f0d1b\": container with ID starting with 83a7d139bd3c213af3d474aad379721792adf314b069e58683759bf8fc6f0d1b not found: ID does not exist"
Jul 06 20:09:09.564176 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:09.564159    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"]
Jul 06 20:09:09.571349 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:09.571330    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-2131ed-84456855b6-f9q2z"]
Jul 06 20:09:09.894520 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:09.894441    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f17545ef-b734-45e3-a0a3-d17b0546f4f3" path="/var/lib/kubelet/pods/f17545ef-b734-45e3-a0a3-d17b0546f4f3/volumes"
Jul 06 20:09:14.099056 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:14.099013    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:14.301224 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:14.301172    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:14.484420 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:14.484336    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:14.584899 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:14.584869    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:14.764036 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:14.764009    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:15.133361 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:15.133285    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:15.234785 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:15.234751    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:15.343268 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:15.343240    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:15.546750 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:15.546713    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"]
Jul 06 20:09:15.546978 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:15.546949    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk" podUID="48837016-f3b3-4b0c-82c7-60936438fa82" containerName="maas-api" containerID="cri-o://f1e05d4b2d1f2ab1e7e75aee1b3ca7c2161ac63baf9ce3dd95013442d9583496" gracePeriod=30
Jul 06 20:09:15.644030 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:15.644006    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:15.810785 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:15.810765    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"
Jul 06 20:09:15.863610 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:15.863586    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48837016-f3b3-4b0c-82c7-60936438fa82-openshift-service-ca\") pod \"48837016-f3b3-4b0c-82c7-60936438fa82\" (UID: \"48837016-f3b3-4b0c-82c7-60936438fa82\") "
Jul 06 20:09:15.863757 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:15.863671    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv4c8\" (UniqueName: \"kubernetes.io/projected/48837016-f3b3-4b0c-82c7-60936438fa82-kube-api-access-fv4c8\") pod \"48837016-f3b3-4b0c-82c7-60936438fa82\" (UID: \"48837016-f3b3-4b0c-82c7-60936438fa82\") "
Jul 06 20:09:15.863757 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:15.863715    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48837016-f3b3-4b0c-82c7-60936438fa82-maas-api-tls\") pod \"48837016-f3b3-4b0c-82c7-60936438fa82\" (UID: \"48837016-f3b3-4b0c-82c7-60936438fa82\") "
Jul 06 20:09:15.863983 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:15.863957    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48837016-f3b3-4b0c-82c7-60936438fa82-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "48837016-f3b3-4b0c-82c7-60936438fa82" (UID: "48837016-f3b3-4b0c-82c7-60936438fa82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:09:15.865740 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:15.865712    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48837016-f3b3-4b0c-82c7-60936438fa82-kube-api-access-fv4c8" (OuterVolumeSpecName: "kube-api-access-fv4c8") pod "48837016-f3b3-4b0c-82c7-60936438fa82" (UID: "48837016-f3b3-4b0c-82c7-60936438fa82"). InnerVolumeSpecName "kube-api-access-fv4c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:09:15.865839 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:15.865816    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48837016-f3b3-4b0c-82c7-60936438fa82-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "48837016-f3b3-4b0c-82c7-60936438fa82" (UID: "48837016-f3b3-4b0c-82c7-60936438fa82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:09:15.964802 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:15.964777    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv4c8\" (UniqueName: \"kubernetes.io/projected/48837016-f3b3-4b0c-82c7-60936438fa82-kube-api-access-fv4c8\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:09:15.964802 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:15.964800    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48837016-f3b3-4b0c-82c7-60936438fa82-maas-api-tls\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:09:15.964936 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:15.964809    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48837016-f3b3-4b0c-82c7-60936438fa82-openshift-service-ca\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:09:16.572140 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:16.572110    2562 generic.go:356] "Generic (PLEG): container finished" podID="48837016-f3b3-4b0c-82c7-60936438fa82" containerID="f1e05d4b2d1f2ab1e7e75aee1b3ca7c2161ac63baf9ce3dd95013442d9583496" exitCode=0
Jul 06 20:09:16.572560 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:16.572158    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"
Jul 06 20:09:16.572560 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:16.572198    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk" event={"ID":"48837016-f3b3-4b0c-82c7-60936438fa82","Type":"ContainerDied","Data":"f1e05d4b2d1f2ab1e7e75aee1b3ca7c2161ac63baf9ce3dd95013442d9583496"}
Jul 06 20:09:16.572560 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:16.572252    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk" event={"ID":"48837016-f3b3-4b0c-82c7-60936438fa82","Type":"ContainerDied","Data":"5a2121b8a85179f29d6dc22a17df11b77782d11f43ac2961d16927b311fbed2f"}
Jul 06 20:09:16.572560 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:16.572281    2562 scope.go:117] "RemoveContainer" containerID="f1e05d4b2d1f2ab1e7e75aee1b3ca7c2161ac63baf9ce3dd95013442d9583496"
Jul 06 20:09:16.580653 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:16.580638    2562 scope.go:117] "RemoveContainer" containerID="f1e05d4b2d1f2ab1e7e75aee1b3ca7c2161ac63baf9ce3dd95013442d9583496"
Jul 06 20:09:16.580921 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:09:16.580903    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e05d4b2d1f2ab1e7e75aee1b3ca7c2161ac63baf9ce3dd95013442d9583496\": container with ID starting with f1e05d4b2d1f2ab1e7e75aee1b3ca7c2161ac63baf9ce3dd95013442d9583496 not found: ID does not exist" containerID="f1e05d4b2d1f2ab1e7e75aee1b3ca7c2161ac63baf9ce3dd95013442d9583496"
Jul 06 20:09:16.580965 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:16.580928    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e05d4b2d1f2ab1e7e75aee1b3ca7c2161ac63baf9ce3dd95013442d9583496"} err="failed to get container status \"f1e05d4b2d1f2ab1e7e75aee1b3ca7c2161ac63baf9ce3dd95013442d9583496\": rpc error: code = NotFound desc = could not find container \"f1e05d4b2d1f2ab1e7e75aee1b3ca7c2161ac63baf9ce3dd95013442d9583496\": container with ID starting with f1e05d4b2d1f2ab1e7e75aee1b3ca7c2161ac63baf9ce3dd95013442d9583496 not found: ID does not exist"
Jul 06 20:09:16.610477 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:16.610452    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"]
Jul 06 20:09:16.622524 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:16.622500    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-78a644-54cbf5c49f-lnclk"]
Jul 06 20:09:17.895448 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:17.895415    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48837016-f3b3-4b0c-82c7-60936438fa82" path="/var/lib/kubelet/pods/48837016-f3b3-4b0c-82c7-60936438fa82/volumes"
Jul 06 20:09:21.042698 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:21.042658    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:21.151862 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:21.151832    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:21.323638 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:21.323560    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:21.521584 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:21.521552    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:21.847978 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:21.847949    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:22.143720 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:22.143640    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:22.247411 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:22.247382    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:25.090670 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.090637    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:25.150070 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.150043    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"]
Jul 06 20:09:25.150355 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.150343    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f17545ef-b734-45e3-a0a3-d17b0546f4f3" containerName="maas-api"
Jul 06 20:09:25.150405 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.150356    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f17545ef-b734-45e3-a0a3-d17b0546f4f3" containerName="maas-api"
Jul 06 20:09:25.150405 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.150367    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48837016-f3b3-4b0c-82c7-60936438fa82" containerName="maas-api"
Jul 06 20:09:25.150405 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.150373    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="48837016-f3b3-4b0c-82c7-60936438fa82" containerName="maas-api"
Jul 06 20:09:25.150544 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.150434    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f17545ef-b734-45e3-a0a3-d17b0546f4f3" containerName="maas-api"
Jul 06 20:09:25.150544 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.150444    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="48837016-f3b3-4b0c-82c7-60936438fa82" containerName="maas-api"
Jul 06 20:09:25.154771 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.154754    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"
Jul 06 20:09:25.157903 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.157885    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-cmcxr\""
Jul 06 20:09:25.158341 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.158326    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-87cbde-serving-cert\""
Jul 06 20:09:25.171889 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.171865    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"]
Jul 06 20:09:25.197844 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.197820    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:25.232023 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.231998    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78mkp\" (UniqueName: \"kubernetes.io/projected/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-kube-api-access-78mkp\") pod \"maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx\" (UID: \"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c\") " pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"
Jul 06 20:09:25.232123 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.232046    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-maas-api-tls\") pod \"maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx\" (UID: \"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c\") " pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"
Jul 06 20:09:25.232123 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.232103    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx\" (UID: \"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c\") " pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"
Jul 06 20:09:25.333241 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.333213    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78mkp\" (UniqueName: \"kubernetes.io/projected/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-kube-api-access-78mkp\") pod \"maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx\" (UID: \"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c\") " pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"
Jul 06 20:09:25.333341 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.333249    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-maas-api-tls\") pod \"maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx\" (UID: \"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c\") " pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"
Jul 06 20:09:25.333341 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.333279    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx\" (UID: \"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c\") " pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"
Jul 06 20:09:25.333417 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:09:25.333364    2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-87cbde-serving-cert: secret "maas-api-e2e-shared-a-87cbde-serving-cert" not found
Jul 06 20:09:25.333455 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:09:25.333447    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-maas-api-tls podName:6e8b8528-fc62-4fda-8e2a-cb8bd75c584c nodeName:}" failed. No retries permitted until 2026-07-06 20:09:25.8334322 +0000 UTC m=+2214.374608854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-maas-api-tls") pod "maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx" (UID: "6e8b8528-fc62-4fda-8e2a-cb8bd75c584c") : secret "maas-api-e2e-shared-a-87cbde-serving-cert" not found
Jul 06 20:09:25.333864 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.333848    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx\" (UID: \"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c\") " pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"
Jul 06 20:09:25.342374 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.342325    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78mkp\" (UniqueName: \"kubernetes.io/projected/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-kube-api-access-78mkp\") pod \"maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx\" (UID: \"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c\") " pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"
Jul 06 20:09:25.499625 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.499599    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:25.797639 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.797608    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:25.837750 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.837725    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-maas-api-tls\") pod \"maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx\" (UID: \"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c\") " pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"
Jul 06 20:09:25.840196 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.840171    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-maas-api-tls\") pod \"maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx\" (UID: \"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c\") " pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"
Jul 06 20:09:25.977309 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:25.977278    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:26.065085 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:26.065027    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"
Jul 06 20:09:26.407347 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:26.407267    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"]
Jul 06 20:09:26.410146 ip-10-0-128-200 kubenswrapper[2562]: W0706 20:09:26.410113    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8b8528_fc62_4fda_8e2a_cb8bd75c584c.slice/crio-03d49081d6cb16b8f26bcaf1c05cbe3c9566647897492656f3d2a88f91162f57 WatchSource:0}: Error finding container 03d49081d6cb16b8f26bcaf1c05cbe3c9566647897492656f3d2a88f91162f57: Status 404 returned error can't find the container with id 03d49081d6cb16b8f26bcaf1c05cbe3c9566647897492656f3d2a88f91162f57
Jul 06 20:09:26.625737 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:26.625700    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx" event={"ID":"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c","Type":"ContainerStarted","Data":"03d49081d6cb16b8f26bcaf1c05cbe3c9566647897492656f3d2a88f91162f57"}
Jul 06 20:09:27.632846 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:27.632804    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx" event={"ID":"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c","Type":"ContainerStarted","Data":"a4f76d09d02f1042106bc168aa01b90fac3f3c0763ad5e27b317fee1420cff26"}
Jul 06 20:09:27.633336 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:27.632952    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"
Jul 06 20:09:27.657917 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:27.657866    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx" podStartSLOduration=2.296244488 podStartE2EDuration="2.657852745s" totalImagesPullingTime="361.608257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:09:27.65676504 +0000 UTC m=+2216.197941717" watchObservedRunningTime="2026-07-06 20:09:27.657852745 +0000 UTC m=+2216.199029420"
Jul 06 20:09:28.662347 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:28.662310    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:29.567651 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:29.567614    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:29.861228 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:29.861130    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:30.148152 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:30.148068    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:30.508359 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:30.508334    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:33.276441 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:33.276404    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:33.399575 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:33.399540    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"]
Jul 06 20:09:33.404347 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:33.404321    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"
Jul 06 20:09:33.407351 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:33.407334    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-332acb-serving-cert\""
Jul 06 20:09:33.410717 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:33.410696    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:33.425191 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:33.425171    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"]
Jul 06 20:09:33.498314 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:33.498291    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9bvh\" (UniqueName: \"kubernetes.io/projected/8fa4996d-9a20-4301-87d8-e071f7e11c19-kube-api-access-v9bvh\") pod \"maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m\" (UID: \"8fa4996d-9a20-4301-87d8-e071f7e11c19\") " pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"
Jul 06 20:09:33.498434 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:33.498324    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa4996d-9a20-4301-87d8-e071f7e11c19-openshift-service-ca\") pod \"maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m\" (UID: \"8fa4996d-9a20-4301-87d8-e071f7e11c19\") " pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"
Jul 06 20:09:33.498434 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:33.498401    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fa4996d-9a20-4301-87d8-e071f7e11c19-maas-api-tls\") pod \"maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m\" (UID: \"8fa4996d-9a20-4301-87d8-e071f7e11c19\") " pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"
Jul 06 20:09:33.528079 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:33.528011    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:33.599303 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:33.599284    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fa4996d-9a20-4301-87d8-e071f7e11c19-maas-api-tls\") pod \"maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m\" (UID: \"8fa4996d-9a20-4301-87d8-e071f7e11c19\") " pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"
Jul 06 20:09:33.599408 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:33.599328    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bvh\" (UniqueName: \"kubernetes.io/projected/8fa4996d-9a20-4301-87d8-e071f7e11c19-kube-api-access-v9bvh\") pod \"maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m\" (UID: \"8fa4996d-9a20-4301-87d8-e071f7e11c19\") " pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"
Jul 06 20:09:33.599408 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:33.599352    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa4996d-9a20-4301-87d8-e071f7e11c19-openshift-service-ca\") pod \"maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m\" (UID: \"8fa4996d-9a20-4301-87d8-e071f7e11c19\") " pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"
Jul 06 20:09:33.599521 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:09:33.599426    2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-332acb-serving-cert: secret "maas-api-e2e-shared-b-332acb-serving-cert" not found
Jul 06 20:09:33.599521 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:09:33.599507    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fa4996d-9a20-4301-87d8-e071f7e11c19-maas-api-tls podName:8fa4996d-9a20-4301-87d8-e071f7e11c19 nodeName:}" failed. No retries permitted until 2026-07-06 20:09:34.099468361 +0000 UTC m=+2222.640645015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8fa4996d-9a20-4301-87d8-e071f7e11c19-maas-api-tls") pod "maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m" (UID: "8fa4996d-9a20-4301-87d8-e071f7e11c19") : secret "maas-api-e2e-shared-b-332acb-serving-cert" not found
Jul 06 20:09:33.599915 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:33.599898    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa4996d-9a20-4301-87d8-e071f7e11c19-openshift-service-ca\") pod \"maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m\" (UID: \"8fa4996d-9a20-4301-87d8-e071f7e11c19\") " pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"
Jul 06 20:09:33.614226 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:33.614199    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bvh\" (UniqueName: \"kubernetes.io/projected/8fa4996d-9a20-4301-87d8-e071f7e11c19-kube-api-access-v9bvh\") pod \"maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m\" (UID: \"8fa4996d-9a20-4301-87d8-e071f7e11c19\") " pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"
Jul 06 20:09:33.642411 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:33.642394    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"
Jul 06 20:09:33.704054 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:33.704028    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:34.075340 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:34.075308    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:34.103689 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:34.103664    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fa4996d-9a20-4301-87d8-e071f7e11c19-maas-api-tls\") pod \"maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m\" (UID: \"8fa4996d-9a20-4301-87d8-e071f7e11c19\") " pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"
Jul 06 20:09:34.105906 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:34.105881    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fa4996d-9a20-4301-87d8-e071f7e11c19-maas-api-tls\") pod \"maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m\" (UID: \"8fa4996d-9a20-4301-87d8-e071f7e11c19\") " pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"
Jul 06 20:09:34.174413 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:34.174389    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:34.314949 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:34.314924    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"
Jul 06 20:09:34.450396 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:34.450342    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"]
Jul 06 20:09:34.452542 ip-10-0-128-200 kubenswrapper[2562]: W0706 20:09:34.452517    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa4996d_9a20_4301_87d8_e071f7e11c19.slice/crio-d61ea82d7f16b0b2b44101613efe626367a98fd3a3aa6c03a1e216b3a30b86a5 WatchSource:0}: Error finding container d61ea82d7f16b0b2b44101613efe626367a98fd3a3aa6c03a1e216b3a30b86a5: Status 404 returned error can't find the container with id d61ea82d7f16b0b2b44101613efe626367a98fd3a3aa6c03a1e216b3a30b86a5
Jul 06 20:09:34.663084 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:34.663012    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m" event={"ID":"8fa4996d-9a20-4301-87d8-e071f7e11c19","Type":"ContainerStarted","Data":"d61ea82d7f16b0b2b44101613efe626367a98fd3a3aa6c03a1e216b3a30b86a5"}
Jul 06 20:09:35.670517 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:35.670454    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m" event={"ID":"8fa4996d-9a20-4301-87d8-e071f7e11c19","Type":"ContainerStarted","Data":"02a54366c8328362d8dd2f3331d0fd32ca17fc54bca9137cfb3be88fe6570c20"}
Jul 06 20:09:35.670952 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:35.670548    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"
Jul 06 20:09:35.694568 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:35.694508    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m" podStartSLOduration=2.347036271 podStartE2EDuration="2.694471795s" totalImagesPullingTime="347.435524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:09:35.693737156 +0000 UTC m=+2224.234913832" watchObservedRunningTime="2026-07-06 20:09:35.694471795 +0000 UTC m=+2224.235648475"
Jul 06 20:09:37.701365 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:37.701327    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:41.679789 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:41.679760    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"
Jul 06 20:09:46.700747 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:46.700718    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:46.854297 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:46.854269    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:46.967180 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:46.967116    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:54.941150 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:54.941116    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:55.075561 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:55.075534    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:09:55.672343 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:09:55.672317    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:10:20.597971 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.597941    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"]
Jul 06 20:10:20.601446 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.601428    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:20.604570 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.604546    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-332acb\"/\"openshift-service-ca.crt\""
Jul 06 20:10:20.604683 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.604553    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-332acb\"/\"default-dockercfg-tgfbr\""
Jul 06 20:10:20.605800 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.605781    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-332acb\"/\"auth-test-model-332acb-kserve-self-signed-certs\""
Jul 06 20:10:20.605906 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.605784    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-332acb\"/\"kube-root-ca.crt\""
Jul 06 20:10:20.629029 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.629009    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"]
Jul 06 20:10:20.650049 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.650031    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-tmp-dir\") pod \"auth-test-model-332acb-kserve-bd7cccbcf-4fmfw\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") " pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:20.650141 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.650055    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-tls-certs\") pod \"auth-test-model-332acb-kserve-bd7cccbcf-4fmfw\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") " pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:20.650141 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.650090    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-dshm\") pod \"auth-test-model-332acb-kserve-bd7cccbcf-4fmfw\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") " pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:20.650213 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.650188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-home\") pod \"auth-test-model-332acb-kserve-bd7cccbcf-4fmfw\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") " pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:20.650252 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.650212    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dxft\" (UniqueName: \"kubernetes.io/projected/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-kube-api-access-6dxft\") pod \"auth-test-model-332acb-kserve-bd7cccbcf-4fmfw\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") " pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:20.650286 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.650245    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-model-cache\") pod \"auth-test-model-332acb-kserve-bd7cccbcf-4fmfw\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") " pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:20.725882 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.725796    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:10:20.751123 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.751099    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-dshm\") pod \"auth-test-model-332acb-kserve-bd7cccbcf-4fmfw\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") " pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:20.751239 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.751145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-home\") pod \"auth-test-model-332acb-kserve-bd7cccbcf-4fmfw\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") " pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:20.751239 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.751163    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dxft\" (UniqueName: \"kubernetes.io/projected/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-kube-api-access-6dxft\") pod \"auth-test-model-332acb-kserve-bd7cccbcf-4fmfw\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") " pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:20.751362 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.751230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-model-cache\") pod \"auth-test-model-332acb-kserve-bd7cccbcf-4fmfw\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") " pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:20.751362 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.751299    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-tmp-dir\") pod \"auth-test-model-332acb-kserve-bd7cccbcf-4fmfw\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") " pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:20.751362 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.751322    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-tls-certs\") pod \"auth-test-model-332acb-kserve-bd7cccbcf-4fmfw\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") " pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:20.751688 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.751665    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-model-cache\") pod \"auth-test-model-332acb-kserve-bd7cccbcf-4fmfw\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") " pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:20.751749 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.751687    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-home\") pod \"auth-test-model-332acb-kserve-bd7cccbcf-4fmfw\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") " pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:20.751749 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.751719    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-tmp-dir\") pod \"auth-test-model-332acb-kserve-bd7cccbcf-4fmfw\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") " pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:20.753443 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.753418    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-dshm\") pod \"auth-test-model-332acb-kserve-bd7cccbcf-4fmfw\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") " pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:20.753607 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.753581    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-tls-certs\") pod \"auth-test-model-332acb-kserve-bd7cccbcf-4fmfw\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") " pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:20.761656 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.761639    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dxft\" (UniqueName: \"kubernetes.io/projected/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-kube-api-access-6dxft\") pod \"auth-test-model-332acb-kserve-bd7cccbcf-4fmfw\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") " pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:20.910460 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:20.910403    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:21.128542 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:21.128515    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"]
Jul 06 20:10:21.130446 ip-10-0-128-200 kubenswrapper[2562]: W0706 20:10:21.130420    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda16b70ea_cb5e_4a13_84f6_25e7485e7b6f.slice/crio-0abb076aedd85b4f4b0af6083cd89faf779b7d66dc2511678622944916fffc9a WatchSource:0}: Error finding container 0abb076aedd85b4f4b0af6083cd89faf779b7d66dc2511678622944916fffc9a: Status 404 returned error can't find the container with id 0abb076aedd85b4f4b0af6083cd89faf779b7d66dc2511678622944916fffc9a
Jul 06 20:10:21.888535 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:21.888475    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw" event={"ID":"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f","Type":"ContainerStarted","Data":"b8fe8bafbf6edf99f3a8a8ebf5639b45a64e8063bb8427bf52fe725127be3738"}
Jul 06 20:10:21.895638 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:21.895609    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw" event={"ID":"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f","Type":"ContainerStarted","Data":"0abb076aedd85b4f4b0af6083cd89faf779b7d66dc2511678622944916fffc9a"}
Jul 06 20:10:21.914231 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:21.914187    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw" podStartSLOduration=1.9141732999999999 podStartE2EDuration="1.9141733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:10:21.912178273 +0000 UTC m=+2270.453354952" watchObservedRunningTime="2026-07-06 20:10:21.9141733 +0000 UTC m=+2270.455349976"
Jul 06 20:10:30.911468 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:30.911428    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:30.912063 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:30.911718    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:30.924248 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:30.924222    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:10:30.939960 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:10:30.939941    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:11:38.260095 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:11:38.260062    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:12:16.922672 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:16.922635    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:12:17.845063 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:17.845028    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"]
Jul 06 20:12:17.884576 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:17.884549    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:12:19.165902 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:19.165872    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:12:20.765900 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:20.765857    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:12:20.893894 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:20.893860    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:12:32.864932 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:32.864877    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw" podUID="a16b70ea-cb5e-4a13-84f6-25e7485e7b6f" containerName="main" containerID="cri-o://b8fe8bafbf6edf99f3a8a8ebf5639b45a64e8063bb8427bf52fe725127be3738" gracePeriod=45
Jul 06 20:12:33.113263 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.113241    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:12:33.256415 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.256396    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-model-cache\") pod \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") "
Jul 06 20:12:33.256559 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.256433    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-home\") pod \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") "
Jul 06 20:12:33.256559 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.256459    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-tmp-dir\") pod \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") "
Jul 06 20:12:33.256696 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.256560    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dxft\" (UniqueName: \"kubernetes.io/projected/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-kube-api-access-6dxft\") pod \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") "
Jul 06 20:12:33.256696 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.256595    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-dshm\") pod \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") "
Jul 06 20:12:33.256696 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.256634    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-tls-certs\") pod \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\" (UID: \"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f\") "
Jul 06 20:12:33.256696 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.256637    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a16b70ea-cb5e-4a13-84f6-25e7485e7b6f" (UID: "a16b70ea-cb5e-4a13-84f6-25e7485e7b6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:12:33.256936 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.256684    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-home" (OuterVolumeSpecName: "home") pod "a16b70ea-cb5e-4a13-84f6-25e7485e7b6f" (UID: "a16b70ea-cb5e-4a13-84f6-25e7485e7b6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:12:33.256936 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.256728    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a16b70ea-cb5e-4a13-84f6-25e7485e7b6f" (UID: "a16b70ea-cb5e-4a13-84f6-25e7485e7b6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:12:33.256936 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.256853    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-model-cache\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:12:33.256936 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.256868    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-home\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:12:33.256936 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.256876    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-tmp-dir\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:12:33.258564 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.258521    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-dshm" (OuterVolumeSpecName: "dshm") pod "a16b70ea-cb5e-4a13-84f6-25e7485e7b6f" (UID: "a16b70ea-cb5e-4a13-84f6-25e7485e7b6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:12:33.258564 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.258547    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a16b70ea-cb5e-4a13-84f6-25e7485e7b6f" (UID: "a16b70ea-cb5e-4a13-84f6-25e7485e7b6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:12:33.258564 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.258550    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-kube-api-access-6dxft" (OuterVolumeSpecName: "kube-api-access-6dxft") pod "a16b70ea-cb5e-4a13-84f6-25e7485e7b6f" (UID: "a16b70ea-cb5e-4a13-84f6-25e7485e7b6f"). InnerVolumeSpecName "kube-api-access-6dxft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:12:33.358127 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.358106    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dxft\" (UniqueName: \"kubernetes.io/projected/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-kube-api-access-6dxft\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:12:33.358127 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.358125    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-dshm\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:12:33.358257 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.358134    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f-tls-certs\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:12:33.505802 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.505774    2562 generic.go:356] "Generic (PLEG): container finished" podID="a16b70ea-cb5e-4a13-84f6-25e7485e7b6f" containerID="b8fe8bafbf6edf99f3a8a8ebf5639b45a64e8063bb8427bf52fe725127be3738" exitCode=0
Jul 06 20:12:33.505911 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.505821    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"
Jul 06 20:12:33.505911 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.505865    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw" event={"ID":"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f","Type":"ContainerDied","Data":"b8fe8bafbf6edf99f3a8a8ebf5639b45a64e8063bb8427bf52fe725127be3738"}
Jul 06 20:12:33.505911 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.505907    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw" event={"ID":"a16b70ea-cb5e-4a13-84f6-25e7485e7b6f","Type":"ContainerDied","Data":"0abb076aedd85b4f4b0af6083cd89faf779b7d66dc2511678622944916fffc9a"}
Jul 06 20:12:33.506070 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.505935    2562 scope.go:117] "RemoveContainer" containerID="b8fe8bafbf6edf99f3a8a8ebf5639b45a64e8063bb8427bf52fe725127be3738"
Jul 06 20:12:33.521159 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.521141    2562 scope.go:117] "RemoveContainer" containerID="b8fe8bafbf6edf99f3a8a8ebf5639b45a64e8063bb8427bf52fe725127be3738"
Jul 06 20:12:33.521412 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:12:33.521392    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8fe8bafbf6edf99f3a8a8ebf5639b45a64e8063bb8427bf52fe725127be3738\": container with ID starting with b8fe8bafbf6edf99f3a8a8ebf5639b45a64e8063bb8427bf52fe725127be3738 not found: ID does not exist" containerID="b8fe8bafbf6edf99f3a8a8ebf5639b45a64e8063bb8427bf52fe725127be3738"
Jul 06 20:12:33.521456 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.521419    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8fe8bafbf6edf99f3a8a8ebf5639b45a64e8063bb8427bf52fe725127be3738"} err="failed to get container status \"b8fe8bafbf6edf99f3a8a8ebf5639b45a64e8063bb8427bf52fe725127be3738\": rpc error: code = NotFound desc = could not find container \"b8fe8bafbf6edf99f3a8a8ebf5639b45a64e8063bb8427bf52fe725127be3738\": container with ID starting with b8fe8bafbf6edf99f3a8a8ebf5639b45a64e8063bb8427bf52fe725127be3738 not found: ID does not exist"
Jul 06 20:12:33.547406 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.547385    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"]
Jul 06 20:12:33.565598 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.565579    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-332acb/auth-test-model-332acb-kserve-bd7cccbcf-4fmfw"]
Jul 06 20:12:33.895524 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:33.895451    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a16b70ea-cb5e-4a13-84f6-25e7485e7b6f" path="/var/lib/kubelet/pods/a16b70ea-cb5e-4a13-84f6-25e7485e7b6f/volumes"
Jul 06 20:12:43.776619 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.776589    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"]
Jul 06 20:12:43.777078 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.776896    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a16b70ea-cb5e-4a13-84f6-25e7485e7b6f" containerName="main"
Jul 06 20:12:43.777078 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.776906    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a16b70ea-cb5e-4a13-84f6-25e7485e7b6f" containerName="main"
Jul 06 20:12:43.777078 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.776973    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a16b70ea-cb5e-4a13-84f6-25e7485e7b6f" containerName="main"
Jul 06 20:12:43.780029 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.780015    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:43.783803 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.783783    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-87cbde\"/\"openshift-service-ca.crt\""
Jul 06 20:12:43.785425 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.785402    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-87cbde\"/\"default-dockercfg-p5b9t\""
Jul 06 20:12:43.785563 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.785474    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-87cbde\"/\"kube-root-ca.crt\""
Jul 06 20:12:43.785631 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.785567    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-87cbde\"/\"sub-test-model-87cbde-kserve-self-signed-certs\""
Jul 06 20:12:43.796161 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.796139    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"]
Jul 06 20:12:43.831034 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.831010    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-model-cache\") pod \"sub-test-model-87cbde-kserve-85cbc65bfc-njpvq\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") " pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:43.831148 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.831070    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-dshm\") pod \"sub-test-model-87cbde-kserve-85cbc65bfc-njpvq\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") " pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:43.831148 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.831106    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e431233-1e06-4930-a8c2-c332e46e1da1-tls-certs\") pod \"sub-test-model-87cbde-kserve-85cbc65bfc-njpvq\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") " pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:43.831235 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.831155    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-home\") pod \"sub-test-model-87cbde-kserve-85cbc65bfc-njpvq\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") " pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:43.831235 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.831171    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt5ts\" (UniqueName: \"kubernetes.io/projected/1e431233-1e06-4930-a8c2-c332e46e1da1-kube-api-access-jt5ts\") pod \"sub-test-model-87cbde-kserve-85cbc65bfc-njpvq\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") " pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:43.831235 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.831199    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-tmp-dir\") pod \"sub-test-model-87cbde-kserve-85cbc65bfc-njpvq\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") " pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:43.871685 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.871659    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:12:43.932172 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.932144    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-dshm\") pod \"sub-test-model-87cbde-kserve-85cbc65bfc-njpvq\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") " pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:43.932283 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.932183    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e431233-1e06-4930-a8c2-c332e46e1da1-tls-certs\") pod \"sub-test-model-87cbde-kserve-85cbc65bfc-njpvq\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") " pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:43.932283 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.932212    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-home\") pod \"sub-test-model-87cbde-kserve-85cbc65bfc-njpvq\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") " pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:43.932283 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.932235    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt5ts\" (UniqueName: \"kubernetes.io/projected/1e431233-1e06-4930-a8c2-c332e46e1da1-kube-api-access-jt5ts\") pod \"sub-test-model-87cbde-kserve-85cbc65bfc-njpvq\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") " pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:43.932435 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.932288    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-tmp-dir\") pod \"sub-test-model-87cbde-kserve-85cbc65bfc-njpvq\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") " pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:43.932435 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.932328    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-model-cache\") pod \"sub-test-model-87cbde-kserve-85cbc65bfc-njpvq\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") " pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:43.932636 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.932612    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-home\") pod \"sub-test-model-87cbde-kserve-85cbc65bfc-njpvq\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") " pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:43.932709 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.932667    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-tmp-dir\") pod \"sub-test-model-87cbde-kserve-85cbc65bfc-njpvq\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") " pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:43.932887 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.932866    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-model-cache\") pod \"sub-test-model-87cbde-kserve-85cbc65bfc-njpvq\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") " pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:43.934341 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.934319    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-dshm\") pod \"sub-test-model-87cbde-kserve-85cbc65bfc-njpvq\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") " pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:43.934966 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.934947    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e431233-1e06-4930-a8c2-c332e46e1da1-tls-certs\") pod \"sub-test-model-87cbde-kserve-85cbc65bfc-njpvq\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") " pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:43.941078 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:43.941056    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt5ts\" (UniqueName: \"kubernetes.io/projected/1e431233-1e06-4930-a8c2-c332e46e1da1-kube-api-access-jt5ts\") pod \"sub-test-model-87cbde-kserve-85cbc65bfc-njpvq\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") " pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:44.088832 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:44.088770    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:44.230380 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:44.230358    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"]
Jul 06 20:12:44.232119 ip-10-0-128-200 kubenswrapper[2562]: W0706 20:12:44.232093    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e431233_1e06_4930_a8c2_c332e46e1da1.slice/crio-3722d3ccf1ff0a7718244ae3f2f475a25ef8fa48346393e3b69495a551f93351 WatchSource:0}: Error finding container 3722d3ccf1ff0a7718244ae3f2f475a25ef8fa48346393e3b69495a551f93351: Status 404 returned error can't find the container with id 3722d3ccf1ff0a7718244ae3f2f475a25ef8fa48346393e3b69495a551f93351
Jul 06 20:12:44.556714 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:44.556680    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq" event={"ID":"1e431233-1e06-4930-a8c2-c332e46e1da1","Type":"ContainerStarted","Data":"5fec27dfa29d20a873b081a4b02780c74963b974f8eb2390e80d0409e27b5882"}
Jul 06 20:12:44.556868 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:44.556725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq" event={"ID":"1e431233-1e06-4930-a8c2-c332e46e1da1","Type":"ContainerStarted","Data":"3722d3ccf1ff0a7718244ae3f2f475a25ef8fa48346393e3b69495a551f93351"}
Jul 06 20:12:44.583699 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:44.583654    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq" podStartSLOduration=1.583637671 podStartE2EDuration="1.583637671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:12:44.582691554 +0000 UTC m=+2413.123868232" watchObservedRunningTime="2026-07-06 20:12:44.583637671 +0000 UTC m=+2413.124814346"
Jul 06 20:12:54.089168 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:54.089092    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:54.089168 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:54.089138    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:54.102742 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:54.102716    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:12:54.611732 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:12:54.611703    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:13:12.107502 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:13:12.107457    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:13:32.186218 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:13:32.186186    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:13:32.285113 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:13:32.285088    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:14:07.294623 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:07.294589    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:14:08.682606 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:08.682573    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:14:10.225700 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:10.225662    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"]
Jul 06 20:14:10.241273 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:10.241243    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:14:11.126597 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:11.126565    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:14:17.660416 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:17.660381    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:14:25.245086 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.245006    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq" podUID="1e431233-1e06-4930-a8c2-c332e46e1da1" containerName="main" containerID="cri-o://5fec27dfa29d20a873b081a4b02780c74963b974f8eb2390e80d0409e27b5882" gracePeriod=45
Jul 06 20:14:25.496032 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.495979    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:14:25.584584 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.584559    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-tmp-dir\") pod \"1e431233-1e06-4930-a8c2-c332e46e1da1\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") "
Jul 06 20:14:25.584697 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.584595    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e431233-1e06-4930-a8c2-c332e46e1da1-tls-certs\") pod \"1e431233-1e06-4930-a8c2-c332e46e1da1\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") "
Jul 06 20:14:25.584697 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.584675    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jt5ts\" (UniqueName: \"kubernetes.io/projected/1e431233-1e06-4930-a8c2-c332e46e1da1-kube-api-access-jt5ts\") pod \"1e431233-1e06-4930-a8c2-c332e46e1da1\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") "
Jul 06 20:14:25.584775 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.584702    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-dshm\") pod \"1e431233-1e06-4930-a8c2-c332e46e1da1\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") "
Jul 06 20:14:25.584856 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.584833    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-model-cache\") pod \"1e431233-1e06-4930-a8c2-c332e46e1da1\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") "
Jul 06 20:14:25.584907 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.584896    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-home\") pod \"1e431233-1e06-4930-a8c2-c332e46e1da1\" (UID: \"1e431233-1e06-4930-a8c2-c332e46e1da1\") "
Jul 06 20:14:25.584952 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.584832    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e431233-1e06-4930-a8c2-c332e46e1da1" (UID: "1e431233-1e06-4930-a8c2-c332e46e1da1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:14:25.585048 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.585026    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e431233-1e06-4930-a8c2-c332e46e1da1" (UID: "1e431233-1e06-4930-a8c2-c332e46e1da1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:14:25.585208 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.585190    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-model-cache\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:14:25.585276 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.585188    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-home" (OuterVolumeSpecName: "home") pod "1e431233-1e06-4930-a8c2-c332e46e1da1" (UID: "1e431233-1e06-4930-a8c2-c332e46e1da1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:14:25.585276 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.585217    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-tmp-dir\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:14:25.586837 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.586814    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-dshm" (OuterVolumeSpecName: "dshm") pod "1e431233-1e06-4930-a8c2-c332e46e1da1" (UID: "1e431233-1e06-4930-a8c2-c332e46e1da1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:14:25.586837 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.586821    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e431233-1e06-4930-a8c2-c332e46e1da1-kube-api-access-jt5ts" (OuterVolumeSpecName: "kube-api-access-jt5ts") pod "1e431233-1e06-4930-a8c2-c332e46e1da1" (UID: "1e431233-1e06-4930-a8c2-c332e46e1da1"). InnerVolumeSpecName "kube-api-access-jt5ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:14:25.586986 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.586911    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e431233-1e06-4930-a8c2-c332e46e1da1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e431233-1e06-4930-a8c2-c332e46e1da1" (UID: "1e431233-1e06-4930-a8c2-c332e46e1da1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:14:25.686452 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.686426    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt5ts\" (UniqueName: \"kubernetes.io/projected/1e431233-1e06-4930-a8c2-c332e46e1da1-kube-api-access-jt5ts\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:14:25.686452 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.686448    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-dshm\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:14:25.686601 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.686463    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e431233-1e06-4930-a8c2-c332e46e1da1-home\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:14:25.686601 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:25.686475    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e431233-1e06-4930-a8c2-c332e46e1da1-tls-certs\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:14:26.026287 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:26.026262    2562 generic.go:356] "Generic (PLEG): container finished" podID="1e431233-1e06-4930-a8c2-c332e46e1da1" containerID="5fec27dfa29d20a873b081a4b02780c74963b974f8eb2390e80d0409e27b5882" exitCode=0
Jul 06 20:14:26.026384 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:26.026302    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"
Jul 06 20:14:26.026384 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:26.026308    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq" event={"ID":"1e431233-1e06-4930-a8c2-c332e46e1da1","Type":"ContainerDied","Data":"5fec27dfa29d20a873b081a4b02780c74963b974f8eb2390e80d0409e27b5882"}
Jul 06 20:14:26.026384 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:26.026342    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq" event={"ID":"1e431233-1e06-4930-a8c2-c332e46e1da1","Type":"ContainerDied","Data":"3722d3ccf1ff0a7718244ae3f2f475a25ef8fa48346393e3b69495a551f93351"}
Jul 06 20:14:26.026384 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:26.026357    2562 scope.go:117] "RemoveContainer" containerID="5fec27dfa29d20a873b081a4b02780c74963b974f8eb2390e80d0409e27b5882"
Jul 06 20:14:26.034300 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:26.034284    2562 scope.go:117] "RemoveContainer" containerID="5fec27dfa29d20a873b081a4b02780c74963b974f8eb2390e80d0409e27b5882"
Jul 06 20:14:26.034563 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:14:26.034543    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fec27dfa29d20a873b081a4b02780c74963b974f8eb2390e80d0409e27b5882\": container with ID starting with 5fec27dfa29d20a873b081a4b02780c74963b974f8eb2390e80d0409e27b5882 not found: ID does not exist" containerID="5fec27dfa29d20a873b081a4b02780c74963b974f8eb2390e80d0409e27b5882"
Jul 06 20:14:26.034613 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:26.034572    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fec27dfa29d20a873b081a4b02780c74963b974f8eb2390e80d0409e27b5882"} err="failed to get container status \"5fec27dfa29d20a873b081a4b02780c74963b974f8eb2390e80d0409e27b5882\": rpc error: code = NotFound desc = could not find container \"5fec27dfa29d20a873b081a4b02780c74963b974f8eb2390e80d0409e27b5882\": container with ID starting with 5fec27dfa29d20a873b081a4b02780c74963b974f8eb2390e80d0409e27b5882 not found: ID does not exist"
Jul 06 20:14:26.049867 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:26.049846    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"]
Jul 06 20:14:26.057777 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:26.057754    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-87cbde/sub-test-model-87cbde-kserve-85cbc65bfc-njpvq"]
Jul 06 20:14:27.895738 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:27.895699    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e431233-1e06-4930-a8c2-c332e46e1da1" path="/var/lib/kubelet/pods/1e431233-1e06-4930-a8c2-c332e46e1da1/volumes"
Jul 06 20:14:45.670198 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.670161    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"]
Jul 06 20:14:45.670669 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.670524    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e431233-1e06-4930-a8c2-c332e46e1da1" containerName="main"
Jul 06 20:14:45.670669 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.670536    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e431233-1e06-4930-a8c2-c332e46e1da1" containerName="main"
Jul 06 20:14:45.670669 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.670594    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e431233-1e06-4930-a8c2-c332e46e1da1" containerName="main"
Jul 06 20:14:45.674326 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.674303    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:45.679666 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.679644    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-332acb\"/\"kube-root-ca.crt\""
Jul 06 20:14:45.679798 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.679681    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-332acb\"/\"openshift-service-ca.crt\""
Jul 06 20:14:45.679798 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.679682    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-332acb\"/\"rate-test-model-332acb-kserve-self-signed-certs\""
Jul 06 20:14:45.679798 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.679702    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-332acb\"/\"default-dockercfg-tgfbr\""
Jul 06 20:14:45.690910 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.690890    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"]
Jul 06 20:14:45.705728 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.705701    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:14:45.739220 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.739197    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-home\") pod \"rate-test-model-332acb-kserve-786c547f5-c2cjz\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") " pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:45.739305 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.739233    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-tmp-dir\") pod \"rate-test-model-332acb-kserve-786c547f5-c2cjz\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") " pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:45.739305 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.739250    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-tls-certs\") pod \"rate-test-model-332acb-kserve-786c547f5-c2cjz\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") " pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:45.739305 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.739282    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-dshm\") pod \"rate-test-model-332acb-kserve-786c547f5-c2cjz\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") " pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:45.739415 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.739342    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk7rg\" (UniqueName: \"kubernetes.io/projected/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-kube-api-access-gk7rg\") pod \"rate-test-model-332acb-kserve-786c547f5-c2cjz\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") " pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:45.739415 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.739367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-model-cache\") pod \"rate-test-model-332acb-kserve-786c547f5-c2cjz\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") " pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:45.840046 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.840015    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-tmp-dir\") pod \"rate-test-model-332acb-kserve-786c547f5-c2cjz\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") " pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:45.840179 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.840055    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-tls-certs\") pod \"rate-test-model-332acb-kserve-786c547f5-c2cjz\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") " pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:45.840179 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.840076    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-dshm\") pod \"rate-test-model-332acb-kserve-786c547f5-c2cjz\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") " pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:45.840179 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.840133    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk7rg\" (UniqueName: \"kubernetes.io/projected/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-kube-api-access-gk7rg\") pod \"rate-test-model-332acb-kserve-786c547f5-c2cjz\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") " pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:45.840179 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.840162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-model-cache\") pod \"rate-test-model-332acb-kserve-786c547f5-c2cjz\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") " pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:45.840395 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.840210    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-home\") pod \"rate-test-model-332acb-kserve-786c547f5-c2cjz\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") " pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:45.840609 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.840586    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-tmp-dir\") pod \"rate-test-model-332acb-kserve-786c547f5-c2cjz\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") " pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:45.841054 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.840958    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-model-cache\") pod \"rate-test-model-332acb-kserve-786c547f5-c2cjz\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") " pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:45.841054 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.841021    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-home\") pod \"rate-test-model-332acb-kserve-786c547f5-c2cjz\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") " pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:45.842659 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.842632    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-dshm\") pod \"rate-test-model-332acb-kserve-786c547f5-c2cjz\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") " pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:45.843039 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.843023    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-tls-certs\") pod \"rate-test-model-332acb-kserve-786c547f5-c2cjz\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") " pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:45.850137 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.850116    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk7rg\" (UniqueName: \"kubernetes.io/projected/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-kube-api-access-gk7rg\") pod \"rate-test-model-332acb-kserve-786c547f5-c2cjz\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") " pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:45.984019 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:45.983963    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:46.121968 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:46.121937    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"]
Jul 06 20:14:46.124774 ip-10-0-128-200 kubenswrapper[2562]: W0706 20:14:46.124747    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67e10e49_73ad_4f78_b6c4_9cf75aa494a9.slice/crio-f6896676cd8c056a8aebfae4693f3dc908fa1b87482602bcb95f75dba848d757 WatchSource:0}: Error finding container f6896676cd8c056a8aebfae4693f3dc908fa1b87482602bcb95f75dba848d757: Status 404 returned error can't find the container with id f6896676cd8c056a8aebfae4693f3dc908fa1b87482602bcb95f75dba848d757
Jul 06 20:14:47.124901 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:47.124857    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz" event={"ID":"67e10e49-73ad-4f78-b6c4-9cf75aa494a9","Type":"ContainerStarted","Data":"72f5ef84fe30ba441c10bf85f200fca44d0774d1c72f6bda04c03b76e4ca6635"}
Jul 06 20:14:47.124901 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:47.124906    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz" event={"ID":"67e10e49-73ad-4f78-b6c4-9cf75aa494a9","Type":"ContainerStarted","Data":"f6896676cd8c056a8aebfae4693f3dc908fa1b87482602bcb95f75dba848d757"}
Jul 06 20:14:47.147851 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:47.147793    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz" podStartSLOduration=2.147766151 podStartE2EDuration="2.147766151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:14:47.147257829 +0000 UTC m=+2535.688434505" watchObservedRunningTime="2026-07-06 20:14:47.147766151 +0000 UTC m=+2535.688942829"
Jul 06 20:14:47.700977 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:47.700948    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:14:47.797338 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:47.797307    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:14:55.889879 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:55.889838    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:14:55.984239 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:55.984210    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:55.984406 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:55.984262    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:55.996610 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:55.996584    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:14:56.000351 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:56.000331    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:14:56.174881 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:14:56.174808    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:15:00.162175 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.162149    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j"]
Jul 06 20:15:00.165536 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.165521    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-332acb-29722815-dhx98"]
Jul 06 20:15:00.165642 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.165624    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j"
Jul 06 20:15:00.169651 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.169632    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722815-qvngw"]
Jul 06 20:15:00.169755 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.169737    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332acb-29722815-dhx98"
Jul 06 20:15:00.172737 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.172719    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722815-qvngw"
Jul 06 20:15:00.177627 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.177607    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j"]
Jul 06 20:15:00.180355 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.180330    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-332acb-29722815-dhx98"]
Jul 06 20:15:00.183940 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.183881    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722815-qvngw"]
Jul 06 20:15:00.241369 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.241348    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pbzk\" (UniqueName: \"kubernetes.io/projected/58a6b984-4676-4a76-8486-ee6b3780a68e-kube-api-access-2pbzk\") pod \"maas-api-key-cleanup-29722815-qvngw\" (UID: \"58a6b984-4676-4a76-8486-ee6b3780a68e\") " pod="opendatahub/maas-api-key-cleanup-29722815-qvngw"
Jul 06 20:15:00.241468 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.241391    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2gzs\" (UniqueName: \"kubernetes.io/projected/5b98d826-b94a-4681-ac32-5611a6ba2bd4-kube-api-access-x2gzs\") pod \"maas-api-key-cleanup-e2e-shared-b-332acb-29722815-dhx98\" (UID: \"5b98d826-b94a-4681-ac32-5611a6ba2bd4\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332acb-29722815-dhx98"
Jul 06 20:15:00.241468 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.241410    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcp9p\" (UniqueName: \"kubernetes.io/projected/5656aed4-47d0-4a1a-9b35-a3771d94675f-kube-api-access-kcp9p\") pod \"maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j\" (UID: \"5656aed4-47d0-4a1a-9b35-a3771d94675f\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j"
Jul 06 20:15:00.341970 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.341946    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2gzs\" (UniqueName: \"kubernetes.io/projected/5b98d826-b94a-4681-ac32-5611a6ba2bd4-kube-api-access-x2gzs\") pod \"maas-api-key-cleanup-e2e-shared-b-332acb-29722815-dhx98\" (UID: \"5b98d826-b94a-4681-ac32-5611a6ba2bd4\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332acb-29722815-dhx98"
Jul 06 20:15:00.342059 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.341974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcp9p\" (UniqueName: \"kubernetes.io/projected/5656aed4-47d0-4a1a-9b35-a3771d94675f-kube-api-access-kcp9p\") pod \"maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j\" (UID: \"5656aed4-47d0-4a1a-9b35-a3771d94675f\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j"
Jul 06 20:15:00.342059 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.342025    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pbzk\" (UniqueName: \"kubernetes.io/projected/58a6b984-4676-4a76-8486-ee6b3780a68e-kube-api-access-2pbzk\") pod \"maas-api-key-cleanup-29722815-qvngw\" (UID: \"58a6b984-4676-4a76-8486-ee6b3780a68e\") " pod="opendatahub/maas-api-key-cleanup-29722815-qvngw"
Jul 06 20:15:00.354202 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.354177    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pbzk\" (UniqueName: \"kubernetes.io/projected/58a6b984-4676-4a76-8486-ee6b3780a68e-kube-api-access-2pbzk\") pod \"maas-api-key-cleanup-29722815-qvngw\" (UID: \"58a6b984-4676-4a76-8486-ee6b3780a68e\") " pod="opendatahub/maas-api-key-cleanup-29722815-qvngw"
Jul 06 20:15:00.354202 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.354192    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcp9p\" (UniqueName: \"kubernetes.io/projected/5656aed4-47d0-4a1a-9b35-a3771d94675f-kube-api-access-kcp9p\") pod \"maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j\" (UID: \"5656aed4-47d0-4a1a-9b35-a3771d94675f\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j"
Jul 06 20:15:00.354325 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.354307    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2gzs\" (UniqueName: \"kubernetes.io/projected/5b98d826-b94a-4681-ac32-5611a6ba2bd4-kube-api-access-x2gzs\") pod \"maas-api-key-cleanup-e2e-shared-b-332acb-29722815-dhx98\" (UID: \"5b98d826-b94a-4681-ac32-5611a6ba2bd4\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332acb-29722815-dhx98"
Jul 06 20:15:00.478529 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.478462    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j"
Jul 06 20:15:00.489014 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.488992    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332acb-29722815-dhx98"
Jul 06 20:15:00.494575 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.494553    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722815-qvngw"
Jul 06 20:15:00.840504 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.840464    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j"]
Jul 06 20:15:00.842346 ip-10-0-128-200 kubenswrapper[2562]: W0706 20:15:00.842312    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5656aed4_47d0_4a1a_9b35_a3771d94675f.slice/crio-c09bdf358427a2b05f0a8684f6d2ae8270fbf849e2fcc4b8e16140e49c6f1304 WatchSource:0}: Error finding container c09bdf358427a2b05f0a8684f6d2ae8270fbf849e2fcc4b8e16140e49c6f1304: Status 404 returned error can't find the container with id c09bdf358427a2b05f0a8684f6d2ae8270fbf849e2fcc4b8e16140e49c6f1304
Jul 06 20:15:00.872364 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.872314    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722815-qvngw"]
Jul 06 20:15:00.874648 ip-10-0-128-200 kubenswrapper[2562]: W0706 20:15:00.874614    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a6b984_4676_4a76_8486_ee6b3780a68e.slice/crio-1ff48cc0bbc1d8c0a07cdf4b24dc7c0d1eda7fcf3f37dda6eee83801fd7cfc43 WatchSource:0}: Error finding container 1ff48cc0bbc1d8c0a07cdf4b24dc7c0d1eda7fcf3f37dda6eee83801fd7cfc43: Status 404 returned error can't find the container with id 1ff48cc0bbc1d8c0a07cdf4b24dc7c0d1eda7fcf3f37dda6eee83801fd7cfc43
Jul 06 20:15:00.875450 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:00.875430    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-332acb-29722815-dhx98"]
Jul 06 20:15:00.876791 ip-10-0-128-200 kubenswrapper[2562]: W0706 20:15:00.876769    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b98d826_b94a_4681_ac32_5611a6ba2bd4.slice/crio-aeacc1b659e4d28ed9318e43964ef12d8d460c0e617643bd3ae3cb0b0a4b675d WatchSource:0}: Error finding container aeacc1b659e4d28ed9318e43964ef12d8d460c0e617643bd3ae3cb0b0a4b675d: Status 404 returned error can't find the container with id aeacc1b659e4d28ed9318e43964ef12d8d460c0e617643bd3ae3cb0b0a4b675d
Jul 06 20:15:01.189901 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:01.189873    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722815-qvngw" event={"ID":"58a6b984-4676-4a76-8486-ee6b3780a68e","Type":"ContainerStarted","Data":"1ff48cc0bbc1d8c0a07cdf4b24dc7c0d1eda7fcf3f37dda6eee83801fd7cfc43"}
Jul 06 20:15:01.191867 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:01.191839    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332acb-29722815-dhx98" event={"ID":"5b98d826-b94a-4681-ac32-5611a6ba2bd4","Type":"ContainerStarted","Data":"aeacc1b659e4d28ed9318e43964ef12d8d460c0e617643bd3ae3cb0b0a4b675d"}
Jul 06 20:15:01.193572 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:01.193537    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j" event={"ID":"5656aed4-47d0-4a1a-9b35-a3771d94675f","Type":"ContainerStarted","Data":"6d0bb313aa65d6cde1eef4e933a248121bdedddb02ad7cc52aab793fc56316b6"}
Jul 06 20:15:01.193664 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:01.193582    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j" event={"ID":"5656aed4-47d0-4a1a-9b35-a3771d94675f","Type":"ContainerStarted","Data":"c09bdf358427a2b05f0a8684f6d2ae8270fbf849e2fcc4b8e16140e49c6f1304"}
Jul 06 20:15:01.217937 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:01.217899    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j" podStartSLOduration=1.21788569 podStartE2EDuration="1.21788569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:01.215904973 +0000 UTC m=+2549.757081641" watchObservedRunningTime="2026-07-06 20:15:01.21788569 +0000 UTC m=+2549.759062365"
Jul 06 20:15:02.200169 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:02.200103    2562 generic.go:356] "Generic (PLEG): container finished" podID="5b98d826-b94a-4681-ac32-5611a6ba2bd4" containerID="42a6726f50acf0855896f896c9e076dbc0f28ae4d475d5e73a30b141fb7d0717" exitCode=0
Jul 06 20:15:02.200571 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:02.200187    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332acb-29722815-dhx98" event={"ID":"5b98d826-b94a-4681-ac32-5611a6ba2bd4","Type":"ContainerDied","Data":"42a6726f50acf0855896f896c9e076dbc0f28ae4d475d5e73a30b141fb7d0717"}
Jul 06 20:15:02.201794 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:02.201776    2562 generic.go:356] "Generic (PLEG): container finished" podID="5656aed4-47d0-4a1a-9b35-a3771d94675f" containerID="6d0bb313aa65d6cde1eef4e933a248121bdedddb02ad7cc52aab793fc56316b6" exitCode=0
Jul 06 20:15:02.201890 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:02.201845    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j" event={"ID":"5656aed4-47d0-4a1a-9b35-a3771d94675f","Type":"ContainerDied","Data":"6d0bb313aa65d6cde1eef4e933a248121bdedddb02ad7cc52aab793fc56316b6"}
Jul 06 20:15:02.203360 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:02.203342    2562 generic.go:356] "Generic (PLEG): container finished" podID="58a6b984-4676-4a76-8486-ee6b3780a68e" containerID="59dadf0e608cd4dff04e286787b0d21069f31e04f0196eae0f8c614488e32eb4" exitCode=0
Jul 06 20:15:02.203423 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:02.203376    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722815-qvngw" event={"ID":"58a6b984-4676-4a76-8486-ee6b3780a68e","Type":"ContainerDied","Data":"59dadf0e608cd4dff04e286787b0d21069f31e04f0196eae0f8c614488e32eb4"}
Jul 06 20:15:03.341652 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:03.341426    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j"
Jul 06 20:15:03.368755 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:03.368734    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722815-qvngw"
Jul 06 20:15:03.402739 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:03.402716    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332acb-29722815-dhx98"
Jul 06 20:15:03.468397 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:03.468339    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcp9p\" (UniqueName: \"kubernetes.io/projected/5656aed4-47d0-4a1a-9b35-a3771d94675f-kube-api-access-kcp9p\") pod \"5656aed4-47d0-4a1a-9b35-a3771d94675f\" (UID: \"5656aed4-47d0-4a1a-9b35-a3771d94675f\") "
Jul 06 20:15:03.468397 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:03.468380    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pbzk\" (UniqueName: \"kubernetes.io/projected/58a6b984-4676-4a76-8486-ee6b3780a68e-kube-api-access-2pbzk\") pod \"58a6b984-4676-4a76-8486-ee6b3780a68e\" (UID: \"58a6b984-4676-4a76-8486-ee6b3780a68e\") "
Jul 06 20:15:03.468553 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:03.468454    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2gzs\" (UniqueName: \"kubernetes.io/projected/5b98d826-b94a-4681-ac32-5611a6ba2bd4-kube-api-access-x2gzs\") pod \"5b98d826-b94a-4681-ac32-5611a6ba2bd4\" (UID: \"5b98d826-b94a-4681-ac32-5611a6ba2bd4\") "
Jul 06 20:15:03.470512 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:03.470465    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b98d826-b94a-4681-ac32-5611a6ba2bd4-kube-api-access-x2gzs" (OuterVolumeSpecName: "kube-api-access-x2gzs") pod "5b98d826-b94a-4681-ac32-5611a6ba2bd4" (UID: "5b98d826-b94a-4681-ac32-5611a6ba2bd4"). InnerVolumeSpecName "kube-api-access-x2gzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:15:03.470608 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:03.470510    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5656aed4-47d0-4a1a-9b35-a3771d94675f-kube-api-access-kcp9p" (OuterVolumeSpecName: "kube-api-access-kcp9p") pod "5656aed4-47d0-4a1a-9b35-a3771d94675f" (UID: "5656aed4-47d0-4a1a-9b35-a3771d94675f"). InnerVolumeSpecName "kube-api-access-kcp9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:15:03.470608 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:03.470465    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a6b984-4676-4a76-8486-ee6b3780a68e-kube-api-access-2pbzk" (OuterVolumeSpecName: "kube-api-access-2pbzk") pod "58a6b984-4676-4a76-8486-ee6b3780a68e" (UID: "58a6b984-4676-4a76-8486-ee6b3780a68e"). InnerVolumeSpecName "kube-api-access-2pbzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:15:03.569386 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:03.569367    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2gzs\" (UniqueName: \"kubernetes.io/projected/5b98d826-b94a-4681-ac32-5611a6ba2bd4-kube-api-access-x2gzs\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:15:03.569386 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:03.569386    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcp9p\" (UniqueName: \"kubernetes.io/projected/5656aed4-47d0-4a1a-9b35-a3771d94675f-kube-api-access-kcp9p\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:15:03.569585 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:03.569395    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pbzk\" (UniqueName: \"kubernetes.io/projected/58a6b984-4676-4a76-8486-ee6b3780a68e-kube-api-access-2pbzk\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:15:04.213539 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:04.213449    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332acb-29722815-dhx98"
Jul 06 20:15:04.213539 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:04.213500    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332acb-29722815-dhx98" event={"ID":"5b98d826-b94a-4681-ac32-5611a6ba2bd4","Type":"ContainerDied","Data":"aeacc1b659e4d28ed9318e43964ef12d8d460c0e617643bd3ae3cb0b0a4b675d"}
Jul 06 20:15:04.213752 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:04.213546    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aeacc1b659e4d28ed9318e43964ef12d8d460c0e617643bd3ae3cb0b0a4b675d"
Jul 06 20:15:04.214759 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:04.214744    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j"
Jul 06 20:15:04.214864 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:04.214776    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j" event={"ID":"5656aed4-47d0-4a1a-9b35-a3771d94675f","Type":"ContainerDied","Data":"c09bdf358427a2b05f0a8684f6d2ae8270fbf849e2fcc4b8e16140e49c6f1304"}
Jul 06 20:15:04.214864 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:04.214819    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c09bdf358427a2b05f0a8684f6d2ae8270fbf849e2fcc4b8e16140e49c6f1304"
Jul 06 20:15:04.216151 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:04.216123    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722815-qvngw" event={"ID":"58a6b984-4676-4a76-8486-ee6b3780a68e","Type":"ContainerDied","Data":"1ff48cc0bbc1d8c0a07cdf4b24dc7c0d1eda7fcf3f37dda6eee83801fd7cfc43"}
Jul 06 20:15:04.216228 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:04.216152    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722815-qvngw"
Jul 06 20:15:04.216228 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:04.216163    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ff48cc0bbc1d8c0a07cdf4b24dc7c0d1eda7fcf3f37dda6eee83801fd7cfc43"
Jul 06 20:15:38.582830 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:38.582797    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:15:39.485548 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:39.485514    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"]
Jul 06 20:15:39.497474 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:39.497442    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:15:40.661230 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:40.661189    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:15:40.922032 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:40.921943    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:15:41.098037 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:41.098007    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:15:41.194865 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:41.194766    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:15:41.440027 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:41.439990    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"]
Jul 06 20:15:41.440248 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:41.440219    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx" podUID="6e8b8528-fc62-4fda-8e2a-cb8bd75c584c" containerName="maas-api" containerID="cri-o://a4f76d09d02f1042106bc168aa01b90fac3f3c0763ad5e27b317fee1420cff26" gracePeriod=30
Jul 06 20:15:41.475451 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:41.475428    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:15:41.494496 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:41.494464    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j"]
Jul 06 20:15:41.503441 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:41.503420    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-87cbde-29722815-wq67j"]
Jul 06 20:15:41.691951 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:41.691930    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"
Jul 06 20:15:41.763156 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:41.763136    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-maas-api-tls\") pod \"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c\" (UID: \"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c\") "
Jul 06 20:15:41.763253 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:41.763187    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-openshift-service-ca\") pod \"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c\" (UID: \"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c\") "
Jul 06 20:15:41.763253 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:41.763206    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78mkp\" (UniqueName: \"kubernetes.io/projected/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-kube-api-access-78mkp\") pod \"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c\" (UID: \"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c\") "
Jul 06 20:15:41.763582 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:41.763560    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6e8b8528-fc62-4fda-8e2a-cb8bd75c584c" (UID: "6e8b8528-fc62-4fda-8e2a-cb8bd75c584c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:15:41.765170 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:41.765149    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6e8b8528-fc62-4fda-8e2a-cb8bd75c584c" (UID: "6e8b8528-fc62-4fda-8e2a-cb8bd75c584c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:15:41.765256 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:41.765198    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-kube-api-access-78mkp" (OuterVolumeSpecName: "kube-api-access-78mkp") pod "6e8b8528-fc62-4fda-8e2a-cb8bd75c584c" (UID: "6e8b8528-fc62-4fda-8e2a-cb8bd75c584c"). InnerVolumeSpecName "kube-api-access-78mkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:15:41.864434 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:41.864412    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-maas-api-tls\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:15:41.864434 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:41.864433    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-openshift-service-ca\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:15:41.864582 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:41.864444    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78mkp\" (UniqueName: \"kubernetes.io/projected/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c-kube-api-access-78mkp\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:15:41.896364 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:41.896340    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5656aed4-47d0-4a1a-9b35-a3771d94675f" path="/var/lib/kubelet/pods/5656aed4-47d0-4a1a-9b35-a3771d94675f/volumes"
Jul 06 20:15:42.403424 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:42.403392    2562 generic.go:356] "Generic (PLEG): container finished" podID="6e8b8528-fc62-4fda-8e2a-cb8bd75c584c" containerID="a4f76d09d02f1042106bc168aa01b90fac3f3c0763ad5e27b317fee1420cff26" exitCode=0
Jul 06 20:15:42.403603 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:42.403439    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx" event={"ID":"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c","Type":"ContainerDied","Data":"a4f76d09d02f1042106bc168aa01b90fac3f3c0763ad5e27b317fee1420cff26"}
Jul 06 20:15:42.403603 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:42.403458    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"
Jul 06 20:15:42.403603 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:42.403530    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx" event={"ID":"6e8b8528-fc62-4fda-8e2a-cb8bd75c584c","Type":"ContainerDied","Data":"03d49081d6cb16b8f26bcaf1c05cbe3c9566647897492656f3d2a88f91162f57"}
Jul 06 20:15:42.403603 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:42.403555    2562 scope.go:117] "RemoveContainer" containerID="a4f76d09d02f1042106bc168aa01b90fac3f3c0763ad5e27b317fee1420cff26"
Jul 06 20:15:42.412085 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:42.412065    2562 scope.go:117] "RemoveContainer" containerID="a4f76d09d02f1042106bc168aa01b90fac3f3c0763ad5e27b317fee1420cff26"
Jul 06 20:15:42.412351 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:15:42.412321    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f76d09d02f1042106bc168aa01b90fac3f3c0763ad5e27b317fee1420cff26\": container with ID starting with a4f76d09d02f1042106bc168aa01b90fac3f3c0763ad5e27b317fee1420cff26 not found: ID does not exist" containerID="a4f76d09d02f1042106bc168aa01b90fac3f3c0763ad5e27b317fee1420cff26"
Jul 06 20:15:42.412400 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:42.412360    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f76d09d02f1042106bc168aa01b90fac3f3c0763ad5e27b317fee1420cff26"} err="failed to get container status \"a4f76d09d02f1042106bc168aa01b90fac3f3c0763ad5e27b317fee1420cff26\": rpc error: code = NotFound desc = could not find container \"a4f76d09d02f1042106bc168aa01b90fac3f3c0763ad5e27b317fee1420cff26\": container with ID starting with a4f76d09d02f1042106bc168aa01b90fac3f3c0763ad5e27b317fee1420cff26 not found: ID does not exist"
Jul 06 20:15:42.427917 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:42.427894    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"]
Jul 06 20:15:42.429519 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:42.429498    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-87cbde-c8ccb47dd-975xx"]
Jul 06 20:15:43.894714 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:43.894677    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e8b8528-fc62-4fda-8e2a-cb8bd75c584c" path="/var/lib/kubelet/pods/6e8b8528-fc62-4fda-8e2a-cb8bd75c584c/volumes"
Jul 06 20:15:46.005598 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:46.005564    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:15:54.506261 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.506173    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz" podUID="67e10e49-73ad-4f78-b6c4-9cf75aa494a9" containerName="main" containerID="cri-o://72f5ef84fe30ba441c10bf85f200fca44d0774d1c72f6bda04c03b76e4ca6635" gracePeriod=45
Jul 06 20:15:54.761449 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.761390    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:15:54.864634 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.864609    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-tmp-dir\") pod \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") "
Jul 06 20:15:54.864742 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.864669    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-model-cache\") pod \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") "
Jul 06 20:15:54.864742 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.864708    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-home\") pod \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") "
Jul 06 20:15:54.864742 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.864737    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk7rg\" (UniqueName: \"kubernetes.io/projected/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-kube-api-access-gk7rg\") pod \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") "
Jul 06 20:15:54.864880 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.864788    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-dshm\") pod \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") "
Jul 06 20:15:54.864880 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.864822    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-tls-certs\") pod \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\" (UID: \"67e10e49-73ad-4f78-b6c4-9cf75aa494a9\") "
Jul 06 20:15:54.864998 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.864899    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67e10e49-73ad-4f78-b6c4-9cf75aa494a9" (UID: "67e10e49-73ad-4f78-b6c4-9cf75aa494a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:15:54.864998 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.864899    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "67e10e49-73ad-4f78-b6c4-9cf75aa494a9" (UID: "67e10e49-73ad-4f78-b6c4-9cf75aa494a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:15:54.864998 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.864962    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-home" (OuterVolumeSpecName: "home") pod "67e10e49-73ad-4f78-b6c4-9cf75aa494a9" (UID: "67e10e49-73ad-4f78-b6c4-9cf75aa494a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:15:54.865158 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.865101    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-tmp-dir\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:15:54.865158 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.865114    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-model-cache\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:15:54.865158 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.865123    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-home\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:15:54.866816 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.866797    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67e10e49-73ad-4f78-b6c4-9cf75aa494a9" (UID: "67e10e49-73ad-4f78-b6c4-9cf75aa494a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:15:54.866878 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.866838    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-dshm" (OuterVolumeSpecName: "dshm") pod "67e10e49-73ad-4f78-b6c4-9cf75aa494a9" (UID: "67e10e49-73ad-4f78-b6c4-9cf75aa494a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 20:15:54.866978 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.866955    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-kube-api-access-gk7rg" (OuterVolumeSpecName: "kube-api-access-gk7rg") pod "67e10e49-73ad-4f78-b6c4-9cf75aa494a9" (UID: "67e10e49-73ad-4f78-b6c4-9cf75aa494a9"). InnerVolumeSpecName "kube-api-access-gk7rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:15:54.966170 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.966131    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-tls-certs\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:15:54.966170 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.966169    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk7rg\" (UniqueName: \"kubernetes.io/projected/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-kube-api-access-gk7rg\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:15:54.966293 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:54.966179    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e10e49-73ad-4f78-b6c4-9cf75aa494a9-dshm\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:15:55.469467 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:55.469432    2562 generic.go:356] "Generic (PLEG): container finished" podID="67e10e49-73ad-4f78-b6c4-9cf75aa494a9" containerID="72f5ef84fe30ba441c10bf85f200fca44d0774d1c72f6bda04c03b76e4ca6635" exitCode=0
Jul 06 20:15:55.469672 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:55.469477    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"
Jul 06 20:15:55.469672 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:55.469522    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz" event={"ID":"67e10e49-73ad-4f78-b6c4-9cf75aa494a9","Type":"ContainerDied","Data":"72f5ef84fe30ba441c10bf85f200fca44d0774d1c72f6bda04c03b76e4ca6635"}
Jul 06 20:15:55.469672 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:55.469578    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz" event={"ID":"67e10e49-73ad-4f78-b6c4-9cf75aa494a9","Type":"ContainerDied","Data":"f6896676cd8c056a8aebfae4693f3dc908fa1b87482602bcb95f75dba848d757"}
Jul 06 20:15:55.469672 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:55.469597    2562 scope.go:117] "RemoveContainer" containerID="72f5ef84fe30ba441c10bf85f200fca44d0774d1c72f6bda04c03b76e4ca6635"
Jul 06 20:15:55.479220 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:55.479203    2562 scope.go:117] "RemoveContainer" containerID="72f5ef84fe30ba441c10bf85f200fca44d0774d1c72f6bda04c03b76e4ca6635"
Jul 06 20:15:55.479519 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:15:55.479495    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f5ef84fe30ba441c10bf85f200fca44d0774d1c72f6bda04c03b76e4ca6635\": container with ID starting with 72f5ef84fe30ba441c10bf85f200fca44d0774d1c72f6bda04c03b76e4ca6635 not found: ID does not exist" containerID="72f5ef84fe30ba441c10bf85f200fca44d0774d1c72f6bda04c03b76e4ca6635"
Jul 06 20:15:55.479596 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:55.479532    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f5ef84fe30ba441c10bf85f200fca44d0774d1c72f6bda04c03b76e4ca6635"} err="failed to get container status \"72f5ef84fe30ba441c10bf85f200fca44d0774d1c72f6bda04c03b76e4ca6635\": rpc error: code = NotFound desc = could not find container \"72f5ef84fe30ba441c10bf85f200fca44d0774d1c72f6bda04c03b76e4ca6635\": container with ID starting with 72f5ef84fe30ba441c10bf85f200fca44d0774d1c72f6bda04c03b76e4ca6635 not found: ID does not exist"
Jul 06 20:15:55.496989 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:55.496966    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"]
Jul 06 20:15:55.501160 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:55.501136    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-332acb/rate-test-model-332acb-kserve-786c547f5-c2cjz"]
Jul 06 20:15:55.895524 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:15:55.895474    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67e10e49-73ad-4f78-b6c4-9cf75aa494a9" path="/var/lib/kubelet/pods/67e10e49-73ad-4f78-b6c4-9cf75aa494a9/volumes"
Jul 06 20:16:24.753704 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:24.753660    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:16:25.178193 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:25.178124    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:16:25.425891 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:25.425861    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:16:25.779401 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:25.779362    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:16:26.215463 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:26.215381    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:16:26.599572 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:26.599543    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:16:26.726039 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:26.726010    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:16:27.014954 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.014921    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"]
Jul 06 20:16:27.015393 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.015156    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m" podUID="8fa4996d-9a20-4301-87d8-e071f7e11c19" containerName="maas-api" containerID="cri-o://02a54366c8328362d8dd2f3331d0fd32ca17fc54bca9137cfb3be88fe6570c20" gracePeriod=30
Jul 06 20:16:27.048103 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.048082    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:16:27.073460 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.073432    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-332acb-29722815-dhx98"]
Jul 06 20:16:27.082272 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.082248    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-332acb-29722815-dhx98"]
Jul 06 20:16:27.248429 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.248403    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"
Jul 06 20:16:27.405087 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.405022    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9bvh\" (UniqueName: \"kubernetes.io/projected/8fa4996d-9a20-4301-87d8-e071f7e11c19-kube-api-access-v9bvh\") pod \"8fa4996d-9a20-4301-87d8-e071f7e11c19\" (UID: \"8fa4996d-9a20-4301-87d8-e071f7e11c19\") "
Jul 06 20:16:27.405087 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.405057    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fa4996d-9a20-4301-87d8-e071f7e11c19-maas-api-tls\") pod \"8fa4996d-9a20-4301-87d8-e071f7e11c19\" (UID: \"8fa4996d-9a20-4301-87d8-e071f7e11c19\") "
Jul 06 20:16:27.405087 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.405076    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa4996d-9a20-4301-87d8-e071f7e11c19-openshift-service-ca\") pod \"8fa4996d-9a20-4301-87d8-e071f7e11c19\" (UID: \"8fa4996d-9a20-4301-87d8-e071f7e11c19\") "
Jul 06 20:16:27.405530 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.405472    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa4996d-9a20-4301-87d8-e071f7e11c19-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8fa4996d-9a20-4301-87d8-e071f7e11c19" (UID: "8fa4996d-9a20-4301-87d8-e071f7e11c19"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 20:16:27.407067 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.407043    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa4996d-9a20-4301-87d8-e071f7e11c19-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8fa4996d-9a20-4301-87d8-e071f7e11c19" (UID: "8fa4996d-9a20-4301-87d8-e071f7e11c19"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 20:16:27.407135 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.407103    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa4996d-9a20-4301-87d8-e071f7e11c19-kube-api-access-v9bvh" (OuterVolumeSpecName: "kube-api-access-v9bvh") pod "8fa4996d-9a20-4301-87d8-e071f7e11c19" (UID: "8fa4996d-9a20-4301-87d8-e071f7e11c19"). InnerVolumeSpecName "kube-api-access-v9bvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 20:16:27.506025 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.505997    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9bvh\" (UniqueName: \"kubernetes.io/projected/8fa4996d-9a20-4301-87d8-e071f7e11c19-kube-api-access-v9bvh\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:16:27.506025 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.506023    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fa4996d-9a20-4301-87d8-e071f7e11c19-maas-api-tls\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:16:27.506164 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.506036    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fa4996d-9a20-4301-87d8-e071f7e11c19-openshift-service-ca\") on node \"ip-10-0-128-200.ec2.internal\" DevicePath \"\""
Jul 06 20:16:27.624248 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.624219    2562 generic.go:356] "Generic (PLEG): container finished" podID="8fa4996d-9a20-4301-87d8-e071f7e11c19" containerID="02a54366c8328362d8dd2f3331d0fd32ca17fc54bca9137cfb3be88fe6570c20" exitCode=0
Jul 06 20:16:27.624367 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.624267    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"
Jul 06 20:16:27.624367 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.624307    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m" event={"ID":"8fa4996d-9a20-4301-87d8-e071f7e11c19","Type":"ContainerDied","Data":"02a54366c8328362d8dd2f3331d0fd32ca17fc54bca9137cfb3be88fe6570c20"}
Jul 06 20:16:27.624367 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.624366    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m" event={"ID":"8fa4996d-9a20-4301-87d8-e071f7e11c19","Type":"ContainerDied","Data":"d61ea82d7f16b0b2b44101613efe626367a98fd3a3aa6c03a1e216b3a30b86a5"}
Jul 06 20:16:27.624556 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.624396    2562 scope.go:117] "RemoveContainer" containerID="02a54366c8328362d8dd2f3331d0fd32ca17fc54bca9137cfb3be88fe6570c20"
Jul 06 20:16:27.632882 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.632862    2562 scope.go:117] "RemoveContainer" containerID="02a54366c8328362d8dd2f3331d0fd32ca17fc54bca9137cfb3be88fe6570c20"
Jul 06 20:16:27.633145 ip-10-0-128-200 kubenswrapper[2562]: E0706 20:16:27.633126    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02a54366c8328362d8dd2f3331d0fd32ca17fc54bca9137cfb3be88fe6570c20\": container with ID starting with 02a54366c8328362d8dd2f3331d0fd32ca17fc54bca9137cfb3be88fe6570c20 not found: ID does not exist" containerID="02a54366c8328362d8dd2f3331d0fd32ca17fc54bca9137cfb3be88fe6570c20"
Jul 06 20:16:27.633190 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.633155    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a54366c8328362d8dd2f3331d0fd32ca17fc54bca9137cfb3be88fe6570c20"} err="failed to get container status \"02a54366c8328362d8dd2f3331d0fd32ca17fc54bca9137cfb3be88fe6570c20\": rpc error: code = NotFound desc = could not find container \"02a54366c8328362d8dd2f3331d0fd32ca17fc54bca9137cfb3be88fe6570c20\": container with ID starting with 02a54366c8328362d8dd2f3331d0fd32ca17fc54bca9137cfb3be88fe6570c20 not found: ID does not exist"
Jul 06 20:16:27.664720 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.664669    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"]
Jul 06 20:16:27.671261 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.671242    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-332acb-765c99d4f8-bbg2m"]
Jul 06 20:16:27.894138 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.894112    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b98d826-b94a-4681-ac32-5611a6ba2bd4" path="/var/lib/kubelet/pods/5b98d826-b94a-4681-ac32-5611a6ba2bd4/volumes"
Jul 06 20:16:27.894512 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:27.894479    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fa4996d-9a20-4301-87d8-e071f7e11c19" path="/var/lib/kubelet/pods/8fa4996d-9a20-4301-87d8-e071f7e11c19/volumes"
Jul 06 20:16:33.880828 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:33.880786    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:16:33.949550 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:33.949523    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:16:34.201321 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:34.201203    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:16:34.463623 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:34.463519    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:16:39.858679 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:39.858643    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zt2qz_f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3/manager/0.log"
Jul 06 20:16:39.984536 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:39.984504    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-cndcc_e4500dcb-07fb-45db-aa5d-e698909e80be/manager/0.log"
Jul 06 20:16:40.113191 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:40.113116    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-6968b6764c-967mv_1da904a7-e89a-450f-9f7d-8c35722909e7/maas-api/0.log"
Jul 06 20:16:40.241864 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:40.241836    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722785-lns4g_20b287a7-6de7-4a7d-b9d8-a286a8fe5e98/cleanup/0.log"
Jul 06 20:16:40.497131 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:40.497070    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722815-qvngw_58a6b984-4676-4a76-8486-ee6b3780a68e/cleanup/0.log"
Jul 06 20:16:40.652476 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:40.652449    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-756d996fff-k5fsp_cb2e423c-8346-4f6d-9d59-90cc9b012ba7/manager/0.log"
Jul 06 20:16:40.947625 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:40.947594    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-7xggj_60dc1c22-9b70-47f5-8e05-81797da863b4/manager/0.log"
Jul 06 20:16:41.218968 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:41.218901    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f6756874-cnbnl_744c1aa4-0e00-4bbf-be88-b6c364363c84/manager/0.log"
Jul 06 20:16:41.228446 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:41.228420    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f6756874-cnbnl_744c1aa4-0e00-4bbf-be88-b6c364363c84/copy-manifests/0.log"
Jul 06 20:16:41.479411 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:41.479331    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-52wzp_05e9136b-d59c-4d3a-8376-46cfe917a1a4/postgres/0.log"
Jul 06 20:16:42.965536 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:42.965473    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-fv64h_7ae24278-0051-4c17-be2d-a240ceebc36c/manager/0.log"
Jul 06 20:16:44.175398 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:44.175365    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-qg5nc_341c3a86-c80d-499e-ad5b-24b5bed6ff49/discovery/0.log"
Jul 06 20:16:44.407467 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:44.407438    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-696fc764fb-fs5fm_f9961d6a-cdc0-40da-a047-8061fb34902f/kube-auth-proxy/0.log"
Jul 06 20:16:44.648834 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:44.648805    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-s8v5b_248c632a-ba2a-40bf-a72e-0610ee5f87bd/payload-pre-processing/0.log"
Jul 06 20:16:45.246285 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:45.246256    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-rk6gs_eb5b2c35-ad8e-46a1-b5ba-bb9a400c4122/main/0.log"
Jul 06 20:16:45.743766 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:45.743731    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-7kxxk_e11e7b03-95ef-4d5a-896c-b433fc344f89/main/0.log"
Jul 06 20:16:45.862421 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:45.862392    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-5glpc_9cd5025e-fac4-414d-93e9-5add82837fe7/main/0.log"
Jul 06 20:16:50.545707 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:50.545666    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:16:50.654862 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:50.654827    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log" limit=1024
Jul 06 20:16:54.071668 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:54.071638    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zgsll_05300884-0a61-4b61-9637-add8ddfff1c5/global-pull-secret-syncer/0.log"
Jul 06 20:16:54.138053 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:54.138031    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cklnx_b3ecf9c2-c7ab-440a-b3b3-3db6bee193ba/konnectivity-agent/0.log"
Jul 06 20:16:54.280820 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:54.280797    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-200.ec2.internal_bc01bcdac46728d38f70a288423f3cc9/haproxy/0.log"
Jul 06 20:16:59.396639 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:16:59.396612    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-fv64h_7ae24278-0051-4c17-be2d-a240ceebc36c/manager/0.log"
Jul 06 20:17:02.392028 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:02.391989    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7ft7v_48ceb622-879a-4b7c-a0c7-364c97ad46b8/node-exporter/0.log"
Jul 06 20:17:02.458676 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:02.458652    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7ft7v_48ceb622-879a-4b7c-a0c7-364c97ad46b8/kube-rbac-proxy/0.log"
Jul 06 20:17:02.551324 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:02.551303    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7ft7v_48ceb622-879a-4b7c-a0c7-364c97ad46b8/init-textfile/0.log"
Jul 06 20:17:03.801224 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.801189    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"]
Jul 06 20:17:03.801626 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.801543    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa4996d-9a20-4301-87d8-e071f7e11c19" containerName="maas-api"
Jul 06 20:17:03.801626 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.801557    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa4996d-9a20-4301-87d8-e071f7e11c19" containerName="maas-api"
Jul 06 20:17:03.801626 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.801567    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b98d826-b94a-4681-ac32-5611a6ba2bd4" containerName="cleanup"
Jul 06 20:17:03.801626 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.801573    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b98d826-b94a-4681-ac32-5611a6ba2bd4" containerName="cleanup"
Jul 06 20:17:03.801626 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.801590    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58a6b984-4676-4a76-8486-ee6b3780a68e" containerName="cleanup"
Jul 06 20:17:03.801626 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.801595    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="58a6b984-4676-4a76-8486-ee6b3780a68e" containerName="cleanup"
Jul 06 20:17:03.801626 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.801602    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e10e49-73ad-4f78-b6c4-9cf75aa494a9" containerName="main"
Jul 06 20:17:03.801626 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.801607    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e10e49-73ad-4f78-b6c4-9cf75aa494a9" containerName="main"
Jul 06 20:17:03.801626 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.801615    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5656aed4-47d0-4a1a-9b35-a3771d94675f" containerName="cleanup"
Jul 06 20:17:03.801626 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.801621    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5656aed4-47d0-4a1a-9b35-a3771d94675f" containerName="cleanup"
Jul 06 20:17:03.801626 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.801629    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e8b8528-fc62-4fda-8e2a-cb8bd75c584c" containerName="maas-api"
Jul 06 20:17:03.801626 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.801634    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e8b8528-fc62-4fda-8e2a-cb8bd75c584c" containerName="maas-api"
Jul 06 20:17:03.802010 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.801690    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5656aed4-47d0-4a1a-9b35-a3771d94675f" containerName="cleanup"
Jul 06 20:17:03.802010 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.801699    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa4996d-9a20-4301-87d8-e071f7e11c19" containerName="maas-api"
Jul 06 20:17:03.802010 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.801705    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e10e49-73ad-4f78-b6c4-9cf75aa494a9" containerName="main"
Jul 06 20:17:03.802010 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.801710    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="58a6b984-4676-4a76-8486-ee6b3780a68e" containerName="cleanup"
Jul 06 20:17:03.802010 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.801716    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b98d826-b94a-4681-ac32-5611a6ba2bd4" containerName="cleanup"
Jul 06 20:17:03.802010 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.801723    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e8b8528-fc62-4fda-8e2a-cb8bd75c584c" containerName="maas-api"
Jul 06 20:17:03.804915 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.804895    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"
Jul 06 20:17:03.809538 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.809515    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4m6vl\"/\"default-dockercfg-j9ks9\""
Jul 06 20:17:03.810082 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.810058    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4m6vl\"/\"openshift-service-ca.crt\""
Jul 06 20:17:03.810190 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.810089    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4m6vl\"/\"kube-root-ca.crt\""
Jul 06 20:17:03.820075 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.820053    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"]
Jul 06 20:17:03.970840 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.970813    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9b7a470b-d00b-4221-a724-9f83e8736166-podres\") pod \"perf-node-gather-daemonset-qncgr\" (UID: \"9b7a470b-d00b-4221-a724-9f83e8736166\") " pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"
Jul 06 20:17:03.970972 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.970852    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b7a470b-d00b-4221-a724-9f83e8736166-sys\") pod \"perf-node-gather-daemonset-qncgr\" (UID: \"9b7a470b-d00b-4221-a724-9f83e8736166\") " pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"
Jul 06 20:17:03.970972 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.970870    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b7a470b-d00b-4221-a724-9f83e8736166-lib-modules\") pod \"perf-node-gather-daemonset-qncgr\" (UID: \"9b7a470b-d00b-4221-a724-9f83e8736166\") " pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"
Jul 06 20:17:03.970972 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.970892    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9b7a470b-d00b-4221-a724-9f83e8736166-proc\") pod \"perf-node-gather-daemonset-qncgr\" (UID: \"9b7a470b-d00b-4221-a724-9f83e8736166\") " pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"
Jul 06 20:17:03.970972 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:03.970947    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8xtz\" (UniqueName: \"kubernetes.io/projected/9b7a470b-d00b-4221-a724-9f83e8736166-kube-api-access-c8xtz\") pod \"perf-node-gather-daemonset-qncgr\" (UID: \"9b7a470b-d00b-4221-a724-9f83e8736166\") " pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"
Jul 06 20:17:04.071860 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:04.071794    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9b7a470b-d00b-4221-a724-9f83e8736166-podres\") pod \"perf-node-gather-daemonset-qncgr\" (UID: \"9b7a470b-d00b-4221-a724-9f83e8736166\") " pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"
Jul 06 20:17:04.071860 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:04.071842    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b7a470b-d00b-4221-a724-9f83e8736166-sys\") pod \"perf-node-gather-daemonset-qncgr\" (UID: \"9b7a470b-d00b-4221-a724-9f83e8736166\") " pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"
Jul 06 20:17:04.071860 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:04.071859    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b7a470b-d00b-4221-a724-9f83e8736166-lib-modules\") pod \"perf-node-gather-daemonset-qncgr\" (UID: \"9b7a470b-d00b-4221-a724-9f83e8736166\") " pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"
Jul 06 20:17:04.072091 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:04.071942    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b7a470b-d00b-4221-a724-9f83e8736166-sys\") pod \"perf-node-gather-daemonset-qncgr\" (UID: \"9b7a470b-d00b-4221-a724-9f83e8736166\") " pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"
Jul 06 20:17:04.072091 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:04.071957    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b7a470b-d00b-4221-a724-9f83e8736166-lib-modules\") pod \"perf-node-gather-daemonset-qncgr\" (UID: \"9b7a470b-d00b-4221-a724-9f83e8736166\") " pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"
Jul 06 20:17:04.072091 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:04.071965    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9b7a470b-d00b-4221-a724-9f83e8736166-podres\") pod \"perf-node-gather-daemonset-qncgr\" (UID: \"9b7a470b-d00b-4221-a724-9f83e8736166\") " pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"
Jul 06 20:17:04.072091 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:04.071983    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9b7a470b-d00b-4221-a724-9f83e8736166-proc\") pod \"perf-node-gather-daemonset-qncgr\" (UID: \"9b7a470b-d00b-4221-a724-9f83e8736166\") " pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"
Jul 06 20:17:04.072091 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:04.072029    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8xtz\" (UniqueName: \"kubernetes.io/projected/9b7a470b-d00b-4221-a724-9f83e8736166-kube-api-access-c8xtz\") pod \"perf-node-gather-daemonset-qncgr\" (UID: \"9b7a470b-d00b-4221-a724-9f83e8736166\") " pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"
Jul 06 20:17:04.072091 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:04.072031    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9b7a470b-d00b-4221-a724-9f83e8736166-proc\") pod \"perf-node-gather-daemonset-qncgr\" (UID: \"9b7a470b-d00b-4221-a724-9f83e8736166\") " pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"
Jul 06 20:17:04.082065 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:04.082045    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8xtz\" (UniqueName: \"kubernetes.io/projected/9b7a470b-d00b-4221-a724-9f83e8736166-kube-api-access-c8xtz\") pod \"perf-node-gather-daemonset-qncgr\" (UID: \"9b7a470b-d00b-4221-a724-9f83e8736166\") " pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"
Jul 06 20:17:04.116119 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:04.116097    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"
Jul 06 20:17:04.248113 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:04.248090    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"]
Jul 06 20:17:04.249910 ip-10-0-128-200 kubenswrapper[2562]: W0706 20:17:04.249877    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9b7a470b_d00b_4221_a724_9f83e8736166.slice/crio-9f3aa451323863f38f58f2ff993184c50864a386201487ffa496b88a6dc8b4ad WatchSource:0}: Error finding container 9f3aa451323863f38f58f2ff993184c50864a386201487ffa496b88a6dc8b4ad: Status 404 returned error can't find the container with id 9f3aa451323863f38f58f2ff993184c50864a386201487ffa496b88a6dc8b4ad
Jul 06 20:17:04.808080 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:04.808047    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr" event={"ID":"9b7a470b-d00b-4221-a724-9f83e8736166","Type":"ContainerStarted","Data":"292d0225dcce9ed55dada427ab2c6c2f3fc4308498623715f1f1d2b9543d2e60"}
Jul 06 20:17:04.808453 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:04.808105    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr" event={"ID":"9b7a470b-d00b-4221-a724-9f83e8736166","Type":"ContainerStarted","Data":"9f3aa451323863f38f58f2ff993184c50864a386201487ffa496b88a6dc8b4ad"}
Jul 06 20:17:04.808453 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:04.808152    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"
Jul 06 20:17:04.831418 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:04.831369    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr" podStartSLOduration=1.83135329 podStartE2EDuration="1.83135329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:17:04.828931643 +0000 UTC m=+2673.370108338" watchObservedRunningTime="2026-07-06 20:17:04.83135329 +0000 UTC m=+2673.372529966"
Jul 06 20:17:08.176149 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:08.176117    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-phrpj_44bd8f04-80fc-4f99-97f3-3f84666ae1d0/dns/0.log"
Jul 06 20:17:08.219018 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:08.218992    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-phrpj_44bd8f04-80fc-4f99-97f3-3f84666ae1d0/kube-rbac-proxy/0.log"
Jul 06 20:17:08.307211 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:08.307187    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n9f6w_5ea2a94f-253e-4bc7-960f-69e371dbbd6a/dns-node-resolver/0.log"
Jul 06 20:17:09.034188 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:09.034151    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2cwdm_75a74486-1003-45bb-9c55-1cbf52ab4090/node-ca/0.log"
Jul 06 20:17:10.403153 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:10.403122    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-qg5nc_341c3a86-c80d-499e-ad5b-24b5bed6ff49/discovery/0.log"
Jul 06 20:17:10.480461 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:10.480441    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-696fc764fb-fs5fm_f9961d6a-cdc0-40da-a047-8061fb34902f/kube-auth-proxy/0.log"
Jul 06 20:17:10.593241 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:10.593191    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-s8v5b_248c632a-ba2a-40bf-a72e-0610ee5f87bd/payload-pre-processing/0.log"
Jul 06 20:17:10.821798 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:10.821771    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4m6vl/perf-node-gather-daemonset-qncgr"
Jul 06 20:17:11.512381 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:11.512351    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n6x2q_3d8e171c-5f31-43a4-ad00-034d98242a89/serve-healthcheck-canary/0.log"
Jul 06 20:17:12.190420 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:12.190388    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-426cv_a71999f1-e3a9-4867-a054-9973c666e32c/kube-rbac-proxy/0.log"
Jul 06 20:17:12.216878 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:12.216857    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-426cv_a71999f1-e3a9-4867-a054-9973c666e32c/exporter/0.log"
Jul 06 20:17:12.245770 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:12.245749    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-426cv_a71999f1-e3a9-4867-a054-9973c666e32c/extractor/0.log"
Jul 06 20:17:14.675278 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:14.675174    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-qg5nc_341c3a86-c80d-499e-ad5b-24b5bed6ff49/discovery/0.log"
Jul 06 20:17:14.711716 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:14.711692    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-696fc764fb-fs5fm_f9961d6a-cdc0-40da-a047-8061fb34902f/kube-auth-proxy/0.log"
Jul 06 20:17:14.784726 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:14.784695    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-s8v5b_248c632a-ba2a-40bf-a72e-0610ee5f87bd/payload-pre-processing/0.log"
Jul 06 20:17:15.200112 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:15.200088    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zt2qz_f71a2a16-e84f-49dc-bdef-bbeb3c9e67f3/manager/0.log"
Jul 06 20:17:15.349465 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:15.349437    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-cndcc_e4500dcb-07fb-45db-aa5d-e698909e80be/manager/0.log"
Jul 06 20:17:15.387687 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:15.387659    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-6968b6764c-967mv_1da904a7-e89a-450f-9f7d-8c35722909e7/maas-api/0.log"
Jul 06 20:17:15.429723 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:15.429664    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722785-lns4g_20b287a7-6de7-4a7d-b9d8-a286a8fe5e98/cleanup/0.log"
Jul 06 20:17:15.504996 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:15.504970    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722815-qvngw_58a6b984-4676-4a76-8486-ee6b3780a68e/cleanup/0.log"
Jul 06 20:17:15.608169 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:15.608145    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-756d996fff-k5fsp_cb2e423c-8346-4f6d-9d59-90cc9b012ba7/manager/0.log"
Jul 06 20:17:15.709093 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:15.709023    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-7xggj_60dc1c22-9b70-47f5-8e05-81797da863b4/manager/0.log"
Jul 06 20:17:15.890884 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:15.890858    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f6756874-cnbnl_744c1aa4-0e00-4bbf-be88-b6c364363c84/manager/0.log"
Jul 06 20:17:15.930474 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:15.930451    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f6756874-cnbnl_744c1aa4-0e00-4bbf-be88-b6c364363c84/copy-manifests/0.log"
Jul 06 20:17:16.032349 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:16.032316    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-52wzp_05e9136b-d59c-4d3a-8376-46cfe917a1a4/postgres/0.log"
Jul 06 20:17:16.117857 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:16.117819    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-qg5nc_341c3a86-c80d-499e-ad5b-24b5bed6ff49/discovery/0.log"
Jul 06 20:17:16.162338 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:16.162310    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-696fc764fb-fs5fm_f9961d6a-cdc0-40da-a047-8061fb34902f/kube-auth-proxy/0.log"
Jul 06 20:17:16.251435 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:16.251412    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-s8v5b_248c632a-ba2a-40bf-a72e-0610ee5f87bd/payload-pre-processing/0.log"
Jul 06 20:17:17.570653 ip-10-0-128-200 kubenswrapper[2562]: I0706 20:17:17.570618    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-s7bn8_6be9df0f-77fa-4213-ab8a-701bb246bf28/openshift-lws-operator/0.log"